Task Dependencies
The edges are the program. What "B runs after A" means when A is skipped, when A is upstream of forty tasks, and when B secretly reads a table nobody declared.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What does "after" actually mean in a task graph — after success, after completion, after data arrival — and which edges are missing from your graph right now?
Downstream tables and their readers. A consumer reading fct_orders is implicitly relying on the graph having run stg_orders first for the same interval; if that edge is not declared, the reliance is a coincidence rather than a guarantee.
The unit is the edge: an ordered pair of tasks for a given interval, plus a rule for what upstream states permit the downstream to start. The rule is part of the edge, and leaving it implicit is how graphs behave surprisingly on bad days.
Wire tasks in the order you wrote them — extract, then transform, then publish — in one long chain. It is easy to read, easy to reason about, and it is exactly right for a pipeline with one source and one output.
A second source appears. Chained sequentially, the second extract now waits for the first for no reason, and the graph's finish time is the sum of two independent things rather than the maximum (Topological Execution).
- A second source appears. Chained sequentially, the second extract now waits for the first for no reason, and the graph's finish time is the sum of two independent things rather than the maximum (Topological Execution).
- One source is optional — a partner feed that is absent some days. Its extract fails, and because the default rule is "all upstreams succeeded", every downstream task including the ones that never needed it is blocked (Partial Failure).
- A cleanup task is added at the end. It runs only on success, so the temporary tables from a failed run are never removed, and the next run reads them.
- A transform is written that reads a table produced by a different DAG. The graph has no edge, so on the day the other DAG runs late, this one reads yesterday's data and succeeds (Stale Dashboards).
- The graph grows to a few hundred tasks and every task depends on one shared early task, so a single flaky extract blocks the whole platform daily and nobody can say which of the four hundred outputs actually needed it (Impact Analysis).
What is actually happening
- A dependency is an edge in a directed acyclic graph, and the executable meaning of the graph is a topological order of it (Topological Sort, DFS Topological Sort). Tasks with no path between them are independent and may run in parallel; the graph is the entire specification of what must precede what.
- Each edge carries a trigger rule: the set of upstream states that permit the downstream to start. The default everywhere is "all upstream succeeded", and it is the right default, but it is one of several and the others exist because real graphs have optional branches and mandatory cleanups.
- Dependencies are evaluated per interval. B for the 11th waits for A for the 11th, not for A in general. Dependence across intervals — this run needs the previous run to have finished — is a different and explicit thing, and it is what makes a task with accumulating state sequential (The High-Water Mark).
- A sensor is a dependency on the world rather than on a task: wait until a file exists, a partition is registered, or an external job reports done. It converts an implicit assumption about timing into an explicit precondition (Ingestion Failure & Recovery).
- Skipping propagates. A skipped task is neither a success nor a failure, and under the default rule its downstream is skipped too — which is usually correct and occasionally means half your platform quietly did nothing while reporting no failures.
- Fan-in is the risky shape. A task with many upstreams starts only when the slowest of them is ready, so its start time is the maximum over its inputs and its failure probability is roughly the union of theirs (Fan-Out: Waiting for the Slowest of Seven).
The graph is the program
Two graphs can contain the same tasks and behave completely differently. The shape decides the finish time, decides how a single failure propagates, and decides which parts of the pipeline can be repaired independently. It is worth drawing before it is worth writing.
The two shapes below hold the same six tasks. The chain finishes in the sum of its task durations and blocks entirely on any failure. The branched version finishes in the length of its longest path and isolates the partner feed, so a partner outage costs the partner-derived model and nothing else.
The second shape also makes the fan-in explicit, and the fan-in is where the reasoning has to happen: fct_orders starts when the *slowest* of its inputs is ready, and it is exposed to the failure of any of them. That is not an argument against fanning in — it is an argument for knowing which of your inputs is genuinely required.
- The chain's finish time is the sum of its tasks; the branched graph's is its longest path. Nothing about the work changed.
- In the chain, a partner-feed failure blocks
stg_orders— a table that does not use the partner feed at all. - The sensor is a dependency on the world. Without it, the dependency still exists; it is just implicit in the schedule and unenforced.
- The cleanup task needs a different rule from every other edge, and getting that wrong means temporary state survives exactly the runs that produced most of it.
Chained by habit — finish time is the sum, any failure blocks everything
extract_orders ─▶ extract_partner ─▶ stg_orders ─▶ stg_partner ─▶ fct_orders ─▶ publish
Branched by data — finish time is the longest path, failures are contained
extract_orders ──▶ stg_orders ────┐
├──▶ fct_orders ──▶ publish
extract_partner ─▶ stg_partner ───┘ ▲
(optional feed) │
wait_for_fx_rates ─────────────────────── ┘
(sensor: external, with a timeout)
cleanup_temp_tables runs when all upstreams have FINISHED,
not when they have SUCCEEDEDWhat "after" means when upstream did not succeed
The default rule — start only when every upstream succeeded — is correct for most edges and quietly wrong for two recurring cases: optional inputs and cleanup. Both are common enough that every mature graph has a few non-default edges, and each one deserves a comment saying why.
Read the table as a set of intentions rather than a set of settings. The question each row answers is "what did you mean by after", and the reason to be explicit is that the default answers it one way for every edge in the graph, including the edges where you meant something else.
The last column is the trap. Every one of these rules has a failure mode that produces no error, because states like skipped and upstream-failed are not failures — they are outcomes the graph considers normal, and a branch that vanishes into skips reports nothing at all.
| You mean | Upstream states that permit the start | Use it for | How it bites |
|---|---|---|---|
| All upstreams succeeded | Every upstream in success | Almost every data edge — the correct default | One optional upstream failing blocks a large subtree that never needed it |
| All upstreams finished | Every upstream terminal: success, failed or skipped | Cleanup, notification, releasing a lock, tearing down a cluster | The task runs on failed runs too, so it must handle the case where its inputs do not exist |
| At least one upstream succeeded | Any one upstream in success | A merge of interchangeable sources where partial input is a valid result | Silently publishes a partial union as though it were the whole thing (Missing Rows) |
| An upstream failed | Any upstream in failed | Compensation, quarantining a bad partition, raising a data incident | Fires on transient failures too, so it must be idempotent or it becomes its own incident |
| Nothing failed, skips allowed | All upstreams success or skipped, none failed | A branch behind a condition that legitimately does not apply today | A cascade of skips looks identical to a healthy quiet day |
| The previous interval finished | This task's own prior run is terminal | Any task with accumulating state — a watermark, a running total, a merge into a shared table | Serialises the task, so a backfill of a year is a year of sequential runs (Incremental Processing) |
Airflow spells these as trigger rules — all_success, all_done, one_success, one_failed, none_failed — and expresses "wait for my own previous run" as a separate task-level flag. Other orchestrators model the same intentions differently and some do not offer all of them. Verify current documentation for names and availability; the intentions are the stable part.
The edges that are not in your graph
Every declared graph sits inside a larger, real graph made of what the code actually reads and writes. Where the two differ, the pipeline is running on coincidence — usually a timing coincidence that has held for months, which is exactly what makes it invisible.
The diagram below shows the standard shape of this problem. A transform in one DAG reads a dimension table produced by another team's DAG. There is no edge, so the scheduler has no opinion about the order; the pipeline is correct only while the other team's job keeps finishing first. On the morning they add a step to their pipeline, yours reads yesterday's dimension, joins successfully, and publishes a fact table whose customer attributes are a day stale.
The fix is not vigilance. It is to make the dependency mechanical: derive edges from the code where the framework allows it, gate on the dataset rather than on the clock where it does not, and treat any read of another team's table as a contract with a stated completion deadline rather than an observation about when it usually lands (Data Contracts).
How to build it
Most important first.
- Declare an edge for every dataset a task reads. The rule is mechanical: if the SQL names a table, there is a dependency, whether or not the graph knows it. Undeclared reads are the most common source of "it worked until it did not".
- Model the graph on data, not on convenience. Two independent sources should be two branches that fan in at the join, not a chain that happens to work.
- Choose the trigger rule explicitly wherever a branch is optional or a task is cleanup. Leaving the default on a cleanup task means cleanup happens only when it was not needed.
- Use a sensor rather than a start-time guess when the input is produced outside your graph. A sensor with a timeout that fails loudly is strictly better than a schedule offset that fails silently (Timeouts).
- Keep the fan-in narrow where you can. A model that depends on eleven upstreams is late whenever any one of them is, and the fix is usually to split the model rather than to speed anything up.
- Express cross-DAG dependencies as real edges. If your platform supports dataset-based triggering, prefer it: an edge on the dataset survives a task rename, and an edge on a task does not (Data Lineage).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The graph guarantees ordering only for the edges you declared, and only within an interval. Everything else — cross-DAG order, cross-interval order, and ordering implied by two tasks touching the same table — is unguaranteed unless stated.
- A satisfied dependency guarantees an upstream *state*, not upstream *correctness*. "A succeeded" means A's process exited zero, so B is guaranteed to start after A did nothing successfully as readily as after A did everything right.
- Independent branches guarantee nothing about each other. If two branches write the same table, the graph will neither prevent it nor warn about it (Reasoning About Races: A Method, Not an Instinct).
- A sensor guarantees that its condition was true at the moment it checked. For a multi-file dataset on object storage, "the first file exists" is not "the dataset is complete" (Object Storage as Data Infrastructure).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check here is a declared-versus-actual dependency audit: parse each task's SQL for the tables it references, compare against the edges declared in the graph, and fail the build on any read that is not an edge. Transformation frameworks that build the graph from the SQL get this for free (dbt Concepts).
- It misses dynamic SQL, tables referenced through views, and reads that happen inside a Python task through a client library rather than in a visible query.
- It also cannot see *write* collisions — two tasks writing the same table are not a read dependency, and only a convention or a catalog of ownership will catch that (Data Ownership).
- The graph's end-to-end latency is its critical path, not the sum of its tasks. Parallel branches cost the maximum of their durations; a chain costs the sum, which is why an accidental chain is a freshness bug.
- Every fan-in inherits the lateness of its slowest input. Adding one slow optional source to a widely-used model makes every consumer of that model as late as the source (The Freshness SLO).
- Sensors trade a predictable finish for a correct one. Consumers who built habits around a fixed completion time experience that trade as a regression, which is a conversation to have before the change rather than after (Data Contracts).
- Adding an edge is safe for correctness and unsafe for freshness: the downstream now waits for something new, and its finish time moves.
- Removing an edge is the reverse and much more dangerous, because it usually happens implicitly — a task is rewritten to read a different table and the old edge is left behind while the new one is never added.
- Renaming a task breaks every edge that referenced it and orphans its run history. Dataset-based edges survive renames because they name the data rather than the job (Metadata: Technical, Operational and Business).
- When re-running, decide the *closure* first: clear the failed task alone, or the failed task and everything reachable downstream from it. The correct answer depends entirely on whether the downstream tasks are idempotent and on whether they already consumed bad data (When a Task Fails Mid-DAG).
- Clearing a task without clearing its downstream leaves consumers holding output derived from data that has since been replaced. That divergence is silent and survives until someone reconciles (Reconciliation).
- For cross-DAG edges, remember that clearing propagates only within the graph that knows about the edge. A downstream DAG in another team's deployment will not be re-triggered by your clear unless the dependency is expressed as data (Impact Analysis).
- Re-running a fan-in point is cheap; re-running its eleven upstreams is not. Establish which inputs actually changed before clearing the whole subtree (Planning a Backfill).
What can go wrong
- An undeclared dependency that works only because of timing, and fails on the first day the timing changes.
- A skip cascading further than intended, so a large branch reports no failures and produces nothing.
- A cleanup task with the default trigger rule, running only when there is nothing to clean up.
- A sensor with no timeout, waiting forever and occupying a worker slot (The Backlog Arithmetic: Four Levers and a Drain Time).
- A cycle introduced by a well-meaning edge, which most tools reject at parse time and some discover at runtime (Deadlock).
- The mitigation failing: a dependency audit that parses SQL, and a task that builds its query as a string at runtime, so the most dangerous read is the one the audit cannot see.
- "The graph shows the dependencies." It shows the ones someone declared. The real dependencies are whatever the code reads, and the gap between those two sets is where incidents live.
- "Sequential is safer." Sequential is slower and no safer for anything the graph does not model. Two tasks in a chain can still race with a third task in another DAG.
- "A skipped task is fine." A skip is a decision that work was unnecessary. When it cascades, it is a decision that a large part of the platform was unnecessary, made by a default nobody chose (Missing Rows).
- "Adding a dependency cannot break anything." It moves every downstream finish time, and freshness is a contract even when it was never written down.
Operating it
- The critical path per run, and how it changes over time. It is the only durable answer to "why is this dataset later than it used to be" (Pipeline Metrics).
- Blocked-task counts by cause — waiting on upstream versus waiting on a worker — which distinguishes a graph problem from a capacity one (Depth Is Not an Emergency; Age Is).
- Skipped task counts. A rising skip rate is a branch quietly disappearing from the pipeline.
- Sensor wait durations, which are the earliest visible signal that an upstream system's own schedule has moved (Freshness Monitoring).
- At 10x tasks, dependency evaluation itself becomes a scheduler cost, and graphs are usually split by ownership rather than by size.
- At 100x, nobody can read the graph, and its value shifts from a picture to a query: what depends on this, what does this depend on, what is the blast radius (Data Lineage).
- Cross-team edges scale worst. A dependency inside a team is a code change; a dependency between teams is a commitment about a completion time, and it needs an owner and a stated deadline (Pipeline SLOs).
- Parallel branches cost concurrency, not total compute: the same work finishes sooner using more slots at once. The bill is shaped by worker capacity rather than by graph structure.
- Sensors cost occupancy for their whole wait, which on a busy platform is a real cost paid to avoid a correctness problem (Cost vs Freshness).
- The expensive dependency mistake is a wide fan-out on a task that is re-run often: every re-run of the shared upstream invites a re-run of everything beneath it (Compute Waste).
- Declaring every dependency makes the graph honest and makes it slower and more brittle in the short term: tasks that used to start optimistically now wait, and edges that were always there become visible as delays.
- Non-default trigger rules buy resilience to optional branches and cost clarity — a graph where several edges have different rules is one a newcomer will misread.
- Sensors buy correctness at the price of predictability and worker occupancy. Fixed schedules buy predictability at the price of being wrong on the worst day.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALGraphs, topological execution and the per-interval evaluation of edges are common to every orchestrator. What differs is whether edges are declared between tasks or inferred from declared datasets, which changes how well the graph survives renames and cross-team boundaries.
- TOOL-SPECIFICTrigger-rule names are Airflow's: all_success, all_done, one_failed, none_failed and their relatives. Dagster expresses the same intent through asset dependencies and run status conditions, and dbt derives edges from the ref graph and has no equivalent of a cleanup-on-failure rule at all.
- ORG-SPECIFICWithin one team, an undeclared dependency is a bug someone can fix in an afternoon. Across teams it is a coordination failure — the upstream owner does not know you read their table, so they are free to reschedule it, and the fix is a contract rather than an edge.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns what it means for two independent branches to write the same table with no coordination, and why "it has always worked" is not evidence of exclusion.
- — DevOps / Production Engineering owns the cross-team version of this problem: a dependency between two teams' pipelines is an interface with an owner and a deadline, and treating it as an observation about timing is the same mistake as depending on an undocumented API.