Airflow Concepts

• A DAG is a Python file that *declares* a graph: tasks and the edges between them. The file is parsed repeatedly by the scheduler, which means it is executed as code on a schedule — top-level work in a DAG file runs constantly, which is a common and expensive surprise.
• The scheduler examines the graph and the metadata database, creates DAG runs for intervals that are due, and marks task instances as ready when their dependencies are satisfied. The metadata database holds every task instance state and is the durable memory that makes any of this more than cron (Scheduler vs Orchestrator).
• An executor hands ready task instances to workers — separate processes, often on separate machines. Task code therefore does not run in the scheduler, does not share memory with it, and cannot pass Python objects to the next task except through an explicit small-value mechanism or through storage (Worker Processes).
• Every DAG run is bound to a logical date and data interval: the window of data the run is responsible for. A daily schedule produces intervals of one day, and the run for an interval is scheduled *after that interval closes* — the run labelled the 11th executes at the start of the 12th, because only then is the 11th complete. Those bounds are exposed to task code as template values, and a task written against them is a function of its interval rather than of the clock, which is exactly the property that makes re-running and backfilling meaningful (Idempotent Data Pipelines).
• Retries are configured on the task, not written into it: an attempt count, a delay, and optionally a per-task timeout. The retry re-runs the same task instance — same interval, new attempt — which is safe if and only if the task is idempotent (Retries in Pipelines).
• Catchup decides what happens to intervals between the DAG's start date and now. Left on, deploying a DAG with an old start date schedules every intervening interval, which is either exactly what you wanted or a self-inflicted load test (Retry Storms: The Load You Generated Yourself).

deployment/
  dags/                  parsed by the scheduler on a loop — keep cheap
    orders_daily.py      declares tasks and edges only
  plugins/
  requirements.txt       must exist on every worker, not just the scheduler

runtime:
  scheduler  ──▶ metadata DB ──▶ executor ──▶ workers ──▶ storage
                     ▲                                      │
                     └──────────── outcomes ────────────────┘

• Pass the interval into every task and never call now() inside one. This single rule is what separates a pipeline that can be repaired from one that cannot, and it costs one templated parameter.
• Write the output to a location keyed by the interval — a partition, a directory, a merge predicate — so a re-run of an interval replaces exactly that interval and touches nothing else (Partitioning).
• Keep DAG files cheap to parse. No database calls, no API calls and no heavy imports at the top level, because the scheduler evaluates that file far more often than the DAG runs.
• Set concurrency limits deliberately: how many runs of this DAG may be active, and how many instances of this task. The defaults are usually permissive, and a backfill is where that becomes visible (Unbounded Concurrency).
• Decide catchup explicitly at creation time rather than discovering the default. Pipelines that must process every interval want it on; pipelines that only care about the present want it off and a documented reason.
• Move data between tasks through storage, not through the orchestrator. The small-value passing mechanism exists for identifiers and counts, and using it for datasets puts your data in the metadata database (The Raw Landing Zone).

• A task instance is executed at least once per attempt boundary: a worker can complete the work and fail to report it, and the scheduler will re-run it. Once-only *effects* are the task's responsibility, never the scheduler's (At-Least-Once Delivery).
• Dependencies are honoured as declared within a DAG for the same interval. Dependencies across DAGs, or across intervals of the same DAG, hold only if you declare them explicitly.
• The interval binding guarantees that the same run identity always refers to the same window of time. It does not guarantee that the *data* for that window was complete when the run executed — that depends on the source (Late-Arriving Data).
• A success state guarantees the task process exited without raising. Nothing more, and this is where most of the domain's incidents live (The Pipeline Succeeded. The Data Is Wrong.).
• Task instances of the same task for different intervals may run concurrently unless limited. Ordering between intervals is not implied by the schedule.