Orchestration
Coordinating work by dependency, state and time. Why a scheduler is not an orchestrator, what a failed task in the middle of a DAG means, and why idempotency is the property that makes re-running safe.
Coordinating work by dependency, state and time — deciding not only when a task may start but whether it should, and what its result means.
"Run at 02:00" versus "run B after A succeeds, retry C, skip D if the source is empty" — a difference in what the system remembers, not in how it is configured.
A DAG of tasks, a scheduler, a metadata database and workers — and the logical data interval, which is the most misunderstood idea in orchestration.
The edges are the program. What "B runs after A" means when A is skipped, when A is upstream of forty tasks, and when B secretly reads a table nobody declared.
A succeeded, B succeeded, C failed. Re-run everything, only C, or C and everything downstream? The answer is decided entirely by idempotency.
Re-running the same logical input must not corrupt or duplicate the result. A pipeline that cannot be re-run is a pipeline whose every bug is permanent.
Process what is new instead of recomputing ten years — and inherit, in exchange, every problem of state: watermarks, late data and two eras in one table.
"Processed through offset X" — the one piece of state that decides what a restart re-reads, and why a log position is a promise and a timestamp is a guess.