Orchestration
Coordinating work by dependency, state and time — deciding not only when a task may start but whether it should, and what its result means.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What actually decides that a task is allowed to run right now, and what does the orchestrator know that a clock does not?
Every downstream table and every human reading one. An analyst does not care which executor ran the transform; they care that fct_orders for yesterday is complete, that it was built from a complete extract, and that if it was not, nothing published it as though it were.
The unit here is the task run: one task, for one logical data interval, on one attempt. Three attempts at the same interval are one unit of work and three rows of state, and confusing those two is how a retried task gets counted as two loads.
A crontab on a box. 0 2 * * * /opt/etl/extract.sh, then 0 3 * * * /opt/etl/transform.sh, with the hour of separation acting as the dependency. It is trivially understandable, it needs no infrastructure, and for a single daily job with one input it is genuinely the right answer.
The extract takes longer than an hour one night because the source was slow. The transform starts anyway at 03:00, reads a half-written directory, succeeds, and publishes a day that is missing its last four hours. Nothing failed (The Pipeline Succeeded. The Data Is Wrong.).
- The extract takes longer than an hour one night because the source was slow. The transform starts anyway at 03:00, reads a half-written directory, succeeds, and publishes a day that is missing its last four hours. Nothing failed (The Pipeline Succeeded. The Data Is Wrong.).
- The extract fails outright. The transform still runs at 03:00, reads yesterday's files because today's are absent, and republishes yesterday's numbers under today's date. The dashboard looks plausible, which is worse than looking broken (Stale Dashboards).
- A run hangs. The next day's run starts on schedule, both processes write to the same output path, and the result is an interleaving neither of them would have produced alone (Atomic Publish).
- Someone re-runs the transform by hand to fix a bug and it appends rather than replaces, doubling every measure for that day. There is no record that the second run happened (Duplicate Rows).
- Six months in there are nineteen cron entries across four machines, the dependency structure lives in the gaps between their start times, and nobody can answer "what feeds this table" without reading shell scripts (Data Lineage).
What is actually happening
- An orchestrator is a state machine over a dependency graph. It stores, durably, the status of every (task, interval) pair — none, queued, running, success, failed, skipped, up-for-retry — and it repeatedly asks a simple question of every task whose interval is due: are all my upstream dependencies in a state that permits me to start?
- That stored state is the whole difference from a scheduler. Time alone cannot answer "did yesterday's load finish", because time does not remember anything. The metadata store does, and that is why the orchestrator can express *conditional* execution and cron cannot (Scheduler vs Orchestrator).
- The dependency structure is a directed acyclic graph, and the execution order is a topological order of it (Topological Execution, DAG (Directed Acyclic Graph)). Tasks with no path between them are independent by construction and may run concurrently; the graph is the only statement of what must precede what.
- The orchestrator separates deciding from doing. A scheduler process evaluates readiness and enqueues work; workers execute it elsewhere and report back. The queue between them is why a slow task does not block scheduling decisions for the rest of the graph (Job Queues).
- Each run is bound to a logical interval — the slice of data it is responsible for — which is deliberately not the same as the wall-clock time it executes at. That binding is what lets a run from three months ago be re-executed today and still mean the same thing (Airflow Concepts).
- Retries, timeouts and concurrency limits are properties of the task definition rather than of the code inside it. Moving them out of the script is what makes them uniform, observable and changeable without a deploy (Retries in Pipelines, Timeouts).
What the orchestrator is actually deciding
Strip away the web UI and the operator library and an orchestrator is a loop. For every task whose logical interval is due, it asks a short sequence of questions, and only if all of them pass does the task become work that a machine somewhere executes. Every question in that sequence is a thing cron cannot ask, because every one of them requires memory of what happened before.
The loop is worth memorising, because almost every orchestration bug is one of these questions being skipped. A task that runs despite a missing input skipped the readiness check. A task that runs twice skipped the concurrency check. A task that runs against today's data when it was asked for March skipped the interval binding — and that one produces no error at all.
Note where the durable state sits. The metadata store is not an implementation detail; it is the component that makes conditional execution possible, and it is also the component whose loss makes the orchestrator forget that yesterday ever happened.
The stages of a run, and what each one promises
A single task run passes through several distinct phases, and they fail in different ways. Treating them as one thing — "the task ran" — is what makes orchestration incidents hard to diagnose, because the symptom of a queueing problem and the symptom of a bad query are both "the table is late".
The guarantee column is the part to read twice. Notice that no stage promises the data is right, and that the only stage which promises anything about *completeness* is the one you have to write yourself. That asymmetry is the reason a validation gate before publish is not an optional refinement.
Notice also that recording the outcome is a separate step from doing the work, and that the gap between them is where at-least-once execution comes from. A worker that finishes writing and then loses its connection has done the work and reported nothing; the orchestrator, correctly, will try again.
- 1Schedule
Decides that an interval is due and creates a run record for it.
guarantees Exactly one run record per (task, interval) unless someone clears it. Not that the run will start soon.
fails by The scheduler process being down, in which case nothing is created, nothing fails, and no alert exists that watches for absence.
- 2Gate
Checks upstream states, sensors and concurrency limits.
guarantees Declared dependencies are satisfied. Says nothing about undeclared reads inside the task.
fails by A sensor that polls for a file that a partial upload created, so presence is confused with completeness.
- 3Queue
Places the run on a queue for an available worker.
guarantees Durability of the intent to run. Not a start time.
fails by Worker starvation — the run is eligible for hours and the dataset is late while every task shows as healthy (The Backlog Arithmetic: Four Levers and a Drain Time).
- 4Execute
Runs the task code with the interval passed in as a parameter.
guarantees Whatever the code guarantees, which by default is nothing.
fails by Reading
now()instead of the interval, which silently converts a re-runnable task into one that can only ever process today. - 5Validate
Asserts row counts, uniqueness and freshness on the produced output.
guarantees Only the assertions written. This is the sole stage that can promise anything about the data.
fails by Not existing, which is the default state of most pipelines (Data Tests).
- 6Publish
Makes the output visible to consumers, ideally as one atomic swap or commit.
guarantees If atomic: readers see the previous version or the new one, never a partial one.
fails by Writing in place, so consumers query a table mid-rewrite and see a day that is genuinely half there (Atomic Publish).
- 7Record
Writes the terminal state back to the metadata store.
guarantees That subsequent scheduling decisions see this outcome.
fails by Dying after the work and before the record, producing a retry of work that already happened — harmless if idempotent, a duplicate otherwise.
Read the failure column as a checklist. Six of these seven failures leave the orchestrator reporting success or reporting nothing at all.
How much orchestration do you actually need
The honest answer for a lot of teams is "less than you have". Orchestration platforms are adopted early because they appear in every reference architecture, and the cost of running one — an upgrade path, a metadata database, a scheduler that can itself be the outage — is paid from day one while the benefit arrives only once dependencies are real.
The trigger for moving up this ladder is never "we should have an orchestrator". It is a specific question you cannot currently answer: did yesterday's load finish, which of these forty tables is stale because of it, and can I re-run March without touching April. Each rung below is the cheapest thing that answers one of those.
The one rung to skip is the middle of the second option. A shell script that greps another script's log file to decide whether to proceed is an orchestrator being written by accident, with no durable state, no retry policy and no history — and it will be maintained by whoever wrote it, forever.
What question about a previous run does this pipeline need to answer before it starts?
when One task, one input, no downstream dependants, and a failure that a human notices the same day.
cost No dependency awareness and no run history. You will find out about failures from a person, and re-running is whatever the script does when invoked twice.
when Two or three steps that always run together and should abort as a unit. The script exits non-zero on the first failure.
cost The whole chain is the retry unit, so a failure in the last step re-runs the expensive first one. Still no history, and still no way to run one interval on demand.
when Multiple producers and consumers, tasks worth retrying independently, and backfills that must be bounded to a range.
cost A platform to operate and upgrade. Dependencies must be declared to be enforced, and the ones you forget are the ones that break.
when The important nouns are datasets rather than jobs, and the question you ask most often is "is this table current" rather than "did this job run".
cost A different mental model that most existing scripts do not fit, and a migration that is not mechanical (dbt Concepts).
when Work should start when data arrives rather than when a clock advances, and arrival is genuinely observable (The Event-Driven Data Platform).
cost No natural interval, so completeness becomes your problem: nothing tells you that all of an hour has arrived, only that something has (Late-Arriving Data).
How to build it
Most important first.
- Declare dependencies explicitly, on data and not on time. "Transform runs after extract succeeds" is a fact about the pipeline; "transform runs at 03:00" is a guess about how long extract takes, and it is wrong on the worst night of the year.
- Bind every run to an explicit interval and pass it into the task as a parameter. A task that computes its own range from
now()cannot be re-run, backfilled or tested, and that single decision is the most common cause of an unfixable pipeline (Idempotent Data Pipelines). - Make every task idempotent before you make anything else clever. Retries, backfills, catch-up and partial-failure recovery are all safe if and only if re-execution is safe (Job Idempotency).
- Keep tasks at a granularity you would want to retry. One monolithic task that extracts, transforms and publishes can only be retried as a whole; three tasks let you re-run the ten-second one instead of the forty-minute one.
- Publish atomically at the end. Write to a location consumers are not reading, validate, then swap or commit — so a failed run leaves no partial output for the next task to consume (Atomic Publish, Data Tests).
- Let the graph be the documentation. If the orchestrator knows the dependencies, it can emit lineage, compute blast radius and answer "what breaks if this is late" without anyone maintaining a diagram (Impact Analysis).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The orchestrator guarantees execution ordering with respect to declared dependencies, and nothing about undeclared ones. A transform that reads a table nobody wired into the graph will happily run before that table is written.
- Task-level delivery is at-least-once in practice: a worker can complete its work and die before recording success, and the orchestrator will then retry it. Any claim of once-only *effects* comes from the task being idempotent, not from the orchestrator (At-Least-Once Delivery, Idempotency).
- A
successstate guarantees the process exited zero. It says nothing about row counts, completeness or correctness — the domain's thesis, restated at the level of the scheduler (The Pipeline Succeeded. The Data Is Wrong.). - Nothing guarantees that two runs of the same task for different intervals do not overlap unless you configure that explicitly; concurrency limits are opt-in, and the default is usually permissive.
- Ordering across separate DAGs is not guaranteed at all unless you express a cross-DAG dependency. Two graphs scheduled at the same time are two independent programs racing on shared tables.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that catches most orchestration failures is a completeness gate before publish: for the interval this run owns, assert the row count is within the historical band for that weekday and that the required upstream partitions exist, and fail the task rather than publishing.
- It misses the case where the upstream itself is complete but wrong, and it misses anything at a grain finer than the partition — a day with the right total and the wrong distribution across customers passes cleanly (Distribution Tests).
- It also cannot see a task that was never scheduled. A DAG that was paused produces no failing runs at all, which is why a freshness check on the *output dataset* has to exist independently of any check inside the pipeline (Freshness Monitoring).
- An orchestrated batch pipeline's freshness floor is the schedule interval plus the critical path through the graph. Adding a task in the middle of that path costs every downstream consumer its duration, every run, forever.
- Dependency-based triggering removes the padding that time-based chaining requires. When the transform waits for the extract rather than for 03:00, a fast night is a fresh morning instead of an idle hour.
- What an orchestrator cannot do is make a consumer's question answerable sooner than the slowest input allows. If one source lands at 06:00, the joined model is a 06:00 model however often the DAG runs (The Freshness SLO).
- Changing the graph changes the meaning of history. Insert a deduplication task today and yesterday's partitions were built by a different program — reproducibility now requires knowing which code version produced which interval (Reprocessing vs Retrying).
- Renaming a task usually loses its state history in the metadata store, so its past runs appear never to have happened. Backfill logic that checks "has this interval already run" then re-runs the entire history.
- Schedule changes are semantic changes. Moving from daily to hourly redefines what one run's interval covers, and every downstream assumption about partition size, watermark spacing and late-data allowance moves with it (Semantic Changes).
- Recovery is re-running a bounded set of (task, interval) pairs. Write down the range explicitly, clear the state for exactly those pairs, and let the graph re-derive downstream — never "re-run the DAG" without a range, which usually means all of history (Planning a Backfill).
- Before re-running anything, answer whether the task overwrites or appends for its interval. Overwrite is repeatable; append is a duplication generator, and the difference decides whether recovery is a command or an incident (Upserts and Merges).
- Re-run into a shadow location and compare against the current output for a period you believe is correct. Publishing a backfill that is wrong in a new way is the standard second incident (Validating a Backfill Before You Publish).
- Keep the raw landing immutable so the graph can be replayed from the beginning if a transformation was wrong for months. If raw was overwritten in place, the recovery ceiling is whatever the source still holds (Keeping Raw History: The Recovery Position and the Liability, The Raw Landing Zone).
What can go wrong
- A task succeeds having done nothing, because its input was empty and the code treats zero rows as a valid day.
- A run overruns and overlaps the next one, and two processes write the same output path (Reasoning About Races: A Method, Not an Instinct).
- A retry re-executes a task whose effects are not idempotent, so the fix for a transient failure creates a permanent data error (Retries).
- The orchestrator itself is down. Nothing fails, nothing runs, and every dashboard quietly shows the last good day.
- Catch-up is enabled on a DAG whose start date is a year ago, and enabling it launches a year of runs at once against a production database (Retry Storms: The Load You Generated Yourself).
- The mitigation fails: an SLA alert exists but fires into a channel nobody reads, so the pipeline is monitored in exactly the sense that satisfies an audit (Alert Fatigue: The Page Nobody Reads).
- "We have Airflow, so we have orchestration." You have a scheduler with a graph in it. If the tasks are not idempotent and the runs are not bound to intervals, you have cron with a nicer interface and worse failure modes.
- "The DAG is green, so the data is good." The DAG is a statement about processes exiting zero. It is structurally unable to observe missing rows, duplicates or wrong logic (Data Quality).
- "Dependencies mean the tasks cannot run at the same time." Dependencies constrain only the paths you declared. Two independent branches run concurrently by design, and if they write the same table that is a race the graph will never warn you about.
- "Orchestration is about scheduling." Scheduling is the least interesting third of it. The state, the retries and the interval semantics are what make re-running safe, and re-running is the whole point (Idempotent Data Pipelines).
Operating it
- Per-task duration over time. The critical path is the only thing that moves end-to-end freshness, and it is visible only as a trend (Pipeline Metrics).
- Queued-to-started delay per task. Rising queue age with flat durations means worker capacity, not slow code (Depth Is Not an Emergency; Age Is).
- Retry counts per task per day. A task that always succeeds on the second attempt is failing every day and reporting success (Pipeline Observability).
- Output dataset freshness, measured against the dataset and not the DAG, so a paused pipeline is still an alert (Freshness Checks).
- At 10x tasks, the scheduler's own loop and its metadata database become the constraint, not the workers. Task count and run history rows grow faster than data volume does.
- At 100x, teams stop running one graph. The unit becomes many small graphs with declared cross-graph dependencies, because a single DAG that everyone edits is a single deploy that everyone blocks (Data Platform Engineering).
- Consumer count changes nothing technically and everything socially: the more teams depend on a schedule, the more a schedule change becomes a negotiation (Data Contracts).
- Orchestration compute is almost always the smallest line in a data platform, and the most frequently optimised, because it is the one with a familiar unit. The cost that matters is what the orchestrator *causes*: how much data each triggered run scans and rewrites (What Actually Drives Data Platform Cost).
- Schedule frequency multiplies every downstream cost linearly. An hourly DAG over a daily dataset does the same total work twenty-four times unless the tasks are genuinely incremental (Incremental Processing).
- Fine-grained tasks cost scheduling overhead and buy retry precision. Thousands of trivial tasks make the metadata store the bottleneck long before the compute is.
- An orchestrator is another production system with its own database, its own upgrades and its own outages. It buys dependency correctness, retries, backfills and lineage; it costs an operational surface that did not exist when the answer was a crontab.
- Declaring dependencies explicitly makes the graph honest and makes it rigid. Every genuine dependency you add is a constraint on how fast the pipeline can possibly be.
- Fine task granularity improves recovery and worsens readability. Somewhere between "one task" and "one task per SQL statement" is a graph a human can hold, and it is closer to the coarse end than most platform diagrams suggest.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe state machine over a dependency graph, and the fact that a success state describes a process rather than a dataset, hold for every orchestrator. What differs between tools is how dependencies are expressed — task graph, asset graph, or data-availability sensor — and how much of the interval semantics is built in.
- SCALE-SPECIFICBelow roughly a handful of tasks with one input each, cron plus a lock file is a defensible answer and an orchestrator is overhead. The inversion happens the first time a task must ask whether another task succeeded, because that question needs durable state that cron does not have.
- TOOL-SPECIFICTask-centric tools such as Airflow schedule task runs and leave data as a side effect; asset-centric tools such as Dagster schedule the materialisation of a declared dataset and derive the tasks. The distinction changes what "the run failed" means: a failed task versus an asset that is now stale.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns how the orchestrator itself is deployed, upgraded and monitored, and how the task code inside it is versioned and rolled back. That domain is being written separately; the orchestrator is an application like any other, and treating it as infrastructure that nobody deploys is how it ends up three major versions behind.
- — Distributed Systems owns why a worker that completes work and dies before acknowledging it cannot be distinguished from one that never did the work. That impossibility is the reason task execution is at-least-once and the reason idempotency is not optional.