Comparisons
Pairs that get conflated in real conversations. Neither column wins — what decides is the requirement, and each record leads with the confusion, because the confusion is the reason the record exists at all.
OLTP vs OLAPBatch vs StreamingETL vs ELTParquet vs AvroLake vs Warehouse vs LakehouseCDC vs PollingEvent modelling vs Snapshot modellingFull refresh vs IncrementalLambda vs KappaOrder grain vs Order-line grain
Full refresh vs Incremental
What people get wrong about this pair
Incremental is treated as the grown-up option. It introduces watermarks, late data and re-run semantics — three new ways to be silently wrong — and a full refresh is genuinely the right answer more often than people admit.
Full refresh
Use it when
The dataset fits in the window, and you would rather have no state to get wrong.
Incremental
Use it when
Rebuilding all history no longer fits, and cost scales with the past instead of with new data.
| Dimension | Full refresh | Incremental |
|---|---|---|
| State to maintain | None | A watermark, and everything that depends on it being right |
| Cost scaling | With all history, every run | With new data |
| Late-arriving data | Handled automatically — it is all recomputed | Must be handled explicitly, or silently lost |
| Recovering from a bug | The next run fixes it | A backfill of the affected range |
| Failure mode | Runs too long, misses its window | Runs fast and quietly skips rows |