IngestionGENERALFORMAT-SPECIFICORG-SPECIFIC

The Raw Landing Zone

Land what arrived, exactly as it arrived, including the fields you do not use. Never clean in place. Partition by arrival so a re-run is bounded.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

When a transformation turns out to have been wrong for six months, what do I re-run it against — and does that thing still exist unmodified?

Who needs this

Not a human. The raw zone's consumer is the staging layer and, occasionally, an engineer during an incident. Analysts should not read it and should be prevented from doing so — a raw dataset has no contract, no stable schema and no deduplication, and any dashboard built on it will break in a way that looks like a data quality problem rather than a boundary violation (Model Layering).

What one row is

One arrival: one file containing the records that landed in one window from one source, exactly as that source presented them. The grain is deliberately not a business entity, because imposing one requires a decision, and the entire purpose of this layer is to defer every decision that could turn out to be wrong (Grain: What Does One Row Represent?).

The obvious build

Skip it. The extract already knows what shape the data should be, so it writes cleaned, typed, deduplicated rows straight into the warehouse. There is one fewer copy, one fewer layer, and the tables are usable the moment they land.

Why it breaks

A transformation bug is found six months later. Fixing it means re-running against the input — but the input was never stored, and the source has since updated, deleted and archived rows. Some of history is repairable and some is simply gone (Reprocessing vs Retrying).

How it breaks with real data

A transformation bug is found six months later. Fixing it means re-running against the input — but the input was never stored, and the source has since updated, deleted and archived rows. Some of history is repairable and some is simply gone (Reprocessing vs Retrying).
A column the extract dropped as "unused" turns out to answer a question the business now cares about. It exists in the source for recent rows only, so the metric can be computed forward and never backward.
A cast to integer silently nulled every amount above a threshold. Because raw was not kept, there is no way to know how many rows were affected or what their values were (Nullability & Defaults).
The source changed the meaning of a status value. The cleaning logic mapped it to the old meaning without erroring, and the only record of what actually arrived is gone (Semantic Changes).
A deduplication applied at ingestion collapsed two genuinely distinct events that shared a payload. Nobody can demonstrate whether they were duplicates, because the evidence was the thing that was removed (Deduplication).
An auditor asks what the source system sent on a specific date. The answer is a reconstruction from a transformed table, which is not evidence (Audit Trails).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

The raw zone exists for exactly one reason: to make every downstream decision reversible. Everything after it is a function of it, so if it is intact and the functions are deterministic, any mistake in any of them is a re-run rather than a loss (Keeping Raw History: The Recovery Position and the Liability).
That property requires immutability, and immutability is a discipline rather than a feature. A zone that is "mostly" append-only, with occasional in-place corrections, provides none of the guarantee: nobody can tell which partitions are original and which were edited, so none of them are evidence.
Partitioning by arrival is what makes a re-run bounded. If a file's path says which window it landed in, then reprocessing a range touches a known set of paths and nothing else, and a re-run replaces its own output rather than mixing with a neighbour's (Partitioning).
Partitioning by *event* time instead makes raw mutable by construction, because a late event for last Tuesday must be written into last Tuesday's partition — which means a partition that consumers may already have read is being changed. Event-time organisation is a modelling concern, and it belongs after this layer (Late-Arriving Data).
Keeping unused fields costs storage and buys optionality. Storage is the cheapest resource in a data platform and the question a business asks next quarter is not knowable now, so the asymmetry favours keeping (What Actually Drives Data Platform Cost).
The zone is also the boundary where the source's schema stops being your problem and starts being a versioned input. Landing self-describing files means a reader can always tell what shape a given partition holds, which is what makes reprocessing across a schema change possible at all (Avro).

The rule, and why it has no exceptions

GENERALThe prefix ordering — source, dataset, arrival date, arrival hour — matters more than the exact names: source first means access and retention policy can be applied per source, and arrival before any event-derived value is what keeps a partition immutable once its interval closes.

The raw landing zone holds what arrived, exactly as it arrived, and nothing ever modifies it. That sentence is the entire design and its value is in the absoluteness rather than in the intent. A zone that is immutable except when somebody needs to fix something provides no guarantee at all, because during an incident nobody can tell which partitions were fixed.

The argument is one of asymmetry. Keeping a field you turn out not to need costs storage, which is the cheapest resource in the platform and is reclaimable at any time. Discarding a field you turn out to need costs the data, permanently, for every row already processed. One direction is a cleanup task; the other is a conversation with the person who asked the question.

The same asymmetry applies to every kind of modification. A cast that turns out to be wrong, a filter that turns out to be too broad, a deduplication rule that turns out to merge distinct events — each is trivially fixable if the input still exists and irreversible if it does not. Deferring all of them costs one extra copy of the data, and the copy is the cheapest insurance in the platform.

The organisation of the zone, by contrast, should be strict. Raw content and raw structure are different things, and confusing them is how a landing zone becomes a bucket of files with names nobody can parse.

Clean on the way in

The ingestion job selects the seven columns anyone currently uses, casts `amount` to a numeric type, drops rows with a null customer id, deduplicates on a payload hash, and writes tidy rows partitioned by order date. The result is immediately usable and there is only one copy.

Land unmodified, partitioned by arrival

The ingestion job writes the source's complete payload — all twenty columns, nulls included, duplicates included — to `raw/<source>/<dataset>/ingest_date=.../ingest_hour=...`, adding only metadata fields that describe the arrival. A separate versioned transformation reads it and produces the tidy table.

The second design can answer questions the first has destroyed the evidence for. Which rows had null customer ids and how many were there? Were those two records genuinely duplicates or distinct events with identical payloads? What did the source actually send on the day the numbers looked wrong? Each of those is a query against raw in the second design and an unanswerable question in the first. The cost is one extra copy of data on the cheapest storage available; the benefit is that every transformation decision remains reversible for as long as the data is retained.

raw/                              <- immutable. nothing here is ever rewritten.
  postgres_orders/                <- one prefix per SOURCE SYSTEM
    orders/                       <- one per DATASET within it
      ingest_date=2026-03-14/     <- ARRIVAL, not event date
        ingest_hour=09/
          part-0000.parquet       <- written once, replaced only by a
          part-0001.parquet          declared re-run of this exact window
          _manifest.json          <- record count, bookmark range, connector
                                     version, content hashes
      ingest_date=2026-03-14/
        ingest_hour=10/
          ...
  stripe_api/
    charges/
      ingest_date=2026-03-14/
        ingest_hour=09/
          part-0000.parquet       <- the API's own document shape, untouched
  clickstream/
    events/
      ingest_date=2026-03-14/
        ingest_hour=09/
          part-0000.parquet

staging/    <- reads raw, writes here. cleaning, casting, dedup live HERE.
marts/      <- reads staging. consumers read THIS.

Properties this layout buys:
  * a re-run of one window touches exactly one prefix
  * a partition becomes immutable the moment its hour ends
  * one source's re-run can never delete another source's data
  * retention and access policy can differ per source prefix
  * "what did Stripe send us at 09:00 on the 14th" is a path, not an investigation

The fields you do not use yet

Every extract faces a moment where someone asks why it is landing twenty columns when the models use seven. The answer is that the models use seven *today*, and the cost of the other thirteen is storage while the cost of not having them is a question that cannot be answered for history.

This is not hypothetical. A field that the source has always populated and nobody has ever needed becomes urgent the moment a business changes — a fulfilment channel, a promotion code, a device type, a partner id. If it was landed, the answer is a transformation change and a backfill. If it was dropped, the answer is that the metric begins on the day someone asked for it.

The schema diff below shows the same upstream change viewed by consumers with and without a raw zone. Note the column that matters: whether the effect is *silent*. A pipeline that drops unknown fields at ingestion absorbs an upstream addition without any signal, which is comfortable right up until the moment the new field is the one you needed.

The genuine counter-argument is governance rather than cost, and it deserves a real answer. Landing everything means landing personal data with no stated purpose. The resolution is a reviewed exclusion list applied at the boundary — specific classified fields excluded deliberately, recorded, and visible — rather than each connector author quietly deciding what looks useful (PII in Pipelines).

Upstream adds `fulfilment_channel` and repurposes `status`

Before

order_id
customer_id
amount_cents
currency
status
created_at
updated_at

After

order_id
customer_id
amount_cents
currency
status
fulfilment_channel
created_at
updated_at

change A new field appears, and status gains a value that previously did not exist. Neither change is announced, and both are backward-compatible in the type sense.

Consumer	Effect	How it shows up
Raw landing zone (land everything)	New partitions contain `fulfilment_channel`; older ones do not. The data itself now records exactly when the change happened, and a future backfill has the field for every row since.	Silently — no error, wrong result
Extract with an explicit column list	The new field is never landed. When it is needed in six months, it exists only for rows extracted after somebody changes the query — history is unavailable and unrecoverable.	Silently — no error, wrong result
Strict loader with a fixed target schema	The load fails on the unexpected column. Loud, disruptive, and — for a raw layer — the wrong behaviour: it converts an upstream addition into an ingestion outage.	Loudly — it raises
Staging model reading raw	Unaffected by the addition, because it selects the columns it needs. This is where a fixed column list belongs: after the data is safely landed.	Silently — no error, wrong result
Any model mapping `status` to a business meaning	The new status value falls through the mapping into a default or a null. The metric changes and nothing errors — the change of meaning is the dangerous half of this diff, not the added column (Semantic Changes).	Silently — no error, wrong result
Incident investigation, six months later	With raw: query the partitions either side of the change and see exactly what arrived and when. Without: reconstruct from a transformed table, which is inference rather than evidence.	Loudly — it raises

Product detail — verify current documentation

Object-storage lifecycle tiers, versioning semantics and the mechanics of deleting a single record from an immutable object differ by provider and change over time — including whether a lifecycle transition is reversible and what retrieval from the coldest tiers involves. Verify current documentation before designing a retention or deletion process around any of it.

Partition by arrival, and keep the re-run bounded

The last piece of the design is the one that makes recovery cheap. If a partition's path names the window it arrived in, then reprocessing a range touches a known, finite set of paths, a re-run replaces its own output, and a partition becomes immutable the moment its interval passes. All three properties come from a naming convention.

Partitioning by event time instead breaks all three at once. A late event for last Tuesday must be written into last Tuesday's partition, which means rewriting a partition consumers may already have read; the set of partitions a re-run touches is unbounded because any window can receive data at any time; and no partition is ever final.

Event-time organisation is genuinely better for reading, and that is what the staging and modelled layers are for. They read arrival partitions with full knowledge of lateness and write event-time ones, which is a transformation with a defined lateness policy rather than an ingestion job silently mutating history (Late Events).

The final convention worth stating: never repair raw. If a source sent something wrong and later sent a correction, land the correction as a new arrival. Both are then evidence, the transformation decides which wins by a source-provided version, and the record of what actually happened survives (Upserts and Merges).

Checks that protect the guarantee, not just the data

Check	Expresses	Catches	Still misses
Content hash per file, compared against the manifest written at landing	Nothing has modified this partition since it was written.	In-place edits, partial overwrites, a re-run that wrote to the wrong path, silent corruption.	A deleted partition — a missing file has no hash to mismatch, which is why continuity is a separate check (Audit Trails).
Arrival-partition continuity per source	Every interval since ingestion began has a partition.	A missed window, a connector writing to a wrong prefix, a lifecycle rule that deleted more than intended.	A partition that exists and is empty for the wrong reason; pair it with a volume comparison against the same interval on prior days.
Writes to closed partitions, counted	The immutability rule is being followed in practice, not just in documentation.	An in-place repair; an event-time partitioning scheme creeping in; a backfill writing outside its declared range.	A declared re-run that legitimately replaces a window — which is why the check should be zero except during recorded recovery, not simply zero.
Reads of raw by principals other than the staging pipeline	The layer boundary is holding.	The first dashboard built directly on raw, months before it becomes a migration problem (Data Platform Anti-Patterns).	A copy someone made into a notebook or a spreadsheet, which leaves the access log looking entirely legitimate.

The third and fourth check the *design*, not the data, which is unusual and deliberate. The value of this layer comes from a rule being followed absolutely, so the rule itself is what needs monitoring.

Reading raw the way downstream should

1-- Raw is queried by ARRIVAL, which is what its partitions name.
2-- Assembling one EVENT day means reading the arrival days it could
3-- have landed in — bounded by the declared lateness allowance.
4 
5CREATE OR REPLACE TABLE staging.orders AS
6SELECT
7    -- typing, renaming and business logic happen HERE, never at ingestion
8    CAST(payload.order_id     AS BIGINT)       AS order_id,
9    CAST(payload.customer_id  AS BIGINT)       AS customer_id,
10    CAST(payload.amount_cents AS BIGINT) / 100 AS amount,
11    payload.currency                           AS currency,
12    payload.status                             AS status_raw,
13    CAST(payload.updated_at   AS TIMESTAMP)    AS source_updated_at,
14    -- metadata the landing zone added, carried forward for debugging
15    _ingest_ts, _source, _bookmark_position
16FROM raw.postgres_orders_orders
17WHERE ingest_date BETWEEN DATE '2026-03-14'      -- the event day
18                      AND DATE '2026-03-16'      -- + lateness allowance
19QUALIFY ROW_NUMBER() OVER (
20    PARTITION BY payload.order_id
21    -- "latest" comes from a SOURCE column, never from arrival order,
22    -- because arrival order is not commit order
23    ORDER BY CAST(payload.updated_at AS TIMESTAMP) DESC,
24             _bookmark_position DESC
25) = 1;
26 
27-- Note what raw did NOT do: no dedup (the QUALIFY does it, here, where the
28-- rule is visible and versioned), no cast (the CAST is here, where a failure
29-- is a code change rather than lost data), no filter on status (the mapping
30-- is downstream, where a new status value can be discovered and handled).

Three things are worth noticing. The lateness allowance appears as an explicit arrival-day range, so the cost of completeness is visible in the query rather than hidden in a partition scheme. Deduplication is expressed as versioned SQL rather than performed irreversibly at ingestion. And "latest" is ordered by a source column with the bookmark position only as a tie-break — arrival order is never the primary sort, because it is not commit order.

How to build it

Most important first.

Write exactly what arrived, byte for byte where practical: the API's JSON document, the CDC event as emitted, the source row with every column. No renames, no casts, no filters, no drops — and never repair raw afterwards: if a source sent something wrong and later sent a correction, land the correction as a new arrival and let the transformation decide which wins, so both remain evidence (ELT: Load First, Transform Where the Data Lives, Upserts and Merges).
Add metadata rather than changing content: arrival timestamp, source identifier, extraction window, offset or bookmark position, connector version, and a file-level record count. These are the fields every incident investigation needs and none of them alter what the source said (Metadata: Technical, Operational and Business).
Partition by arrival — source, then dataset, then arrival date, then hour if volume warrants — so that a partition becomes immutable once its interval passes and a re-run is bounded to a known set of paths. Because the path is a function of the unit of work, re-running a window replaces its own output — the same idempotency convention as everywhere in this module, and the place where it pays off most (Partition Cardinality, Idempotent Data Pipelines).
Enforce read-only access for everyone except the staging layer. This is a permissions decision, not a documentation one — a raw table that analysts *can* query will eventually appear in a dashboard (Data Access Control).
Prefer a self-describing format that carries its own schema over headerless text, so a partition written under an older schema is still readable years later without archaeology (Parquet, CSV, JSON and Their Limits).
Set retention deliberately and separately from the rest of the platform. Raw is simultaneously the strongest recovery position and the largest privacy liability, and those two arguments meet here rather than anywhere else (Data Retention).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

Immutability: what was written stays written and unmodified. This is the only guarantee the layer makes and every other value it has depends on it holding absolutely rather than usually.
Completeness relative to arrival: everything that arrived is here. Not everything that happened — that is a statement about ingestion, measured by reconciliation against the source, and the raw zone cannot make it (Reconciliation).
No deduplication: raw contains duplicates and this is correct. Removing them here would destroy the evidence that redelivery occurred and would apply a deduplication rule that may later prove wrong (Duplicate Rows).
No ordering: files land in arrival order, which is not event order and not commit order. Any ordering a downstream model needs must come from a source-provided column (Event Time).
Schema: whatever the source sent, varying across partitions. Raw guarantees readability, not uniformity — and a layer that guaranteed uniformity would have had to change something (Schema Evolution).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

Assert immutability directly: object versioning or a manifest of content hashes per partition, checked periodically. A raw zone that nobody has verified is unchanged is a raw zone that somebody has changed (Audit Trails).
Assert that each partition is readable and its record count matches the count the ingestion run reported. A file that landed truncated parses cleanly in text formats and fails loudly in self-describing ones — this check closes that gap (Data Tests).
Assert arrival-partition continuity: every expected partition exists for every source. It is the same continuity check as batch windows, applied at the storage layer where it also catches a connector writing to the wrong path.
What none of these can tell you is whether the content is correct. Raw is faithful by definition — it is a faithful copy of whatever the source sent, including whatever the source got wrong (The Pipeline Succeeded. The Data Is Wrong.).

Freshness

Raw is the freshest copy in the platform, because it is the first place data lands. That is a property nobody should build on: it has no contract, and a consumer attracted by its freshness will be broken by its schema.
Its freshness is bounded by the ingestion cadence and nothing else. Batch gives a sawtooth; streaming gives a buffer interval. Neither is improved by anything happening in this layer.
The staleness that matters here is different: how long until a partition is complete. An hour's arrival partition is complete when the hour ends, and knowing that boundary is what lets downstream models process it safely rather than repeatedly (Watermarks).
Landing files at very short intervals to improve freshness produces small files that make every subsequent read slower. The freshness gain accrues to nobody and the cost is paid by every query (File Size and the Small-Files Problem).

When the schema or meaning changes

Raw absorbs schema change rather than resisting it. A new field appears in later partitions and older ones do not have it, which is exactly the record you want: the data itself shows when the change happened (Schema Evolution).
This makes reprocessing across a schema boundary a real design problem rather than an impossible one. A transformation reading a range that spans the change must handle both shapes, and it can, because both shapes are still there (Backward Compatibility).
A field *removed* upstream stops appearing in new partitions and remains in old ones. Downstream code that assumed presence breaks at the boundary, loudly, which is the correct outcome — the alternative is a default value silently filling in (Nullability & Defaults).
What raw cannot record is a change of *meaning* with no change of shape. That requires a contract and a version stamped by the producer, and it is the one evolution problem this layer does not solve (Data Contracts).

How to re-run this safely

The raw zone is the recovery position for everything downstream of it. If it is intact and the transformations are deterministic, every bug in every model is a bounded re-run (Planning a Backfill).
Recovery *of* raw is a different matter: it has no upstream except the source, so if a partition is deleted the only path back is re-extraction, with all the caveats about a source that has moved on (What Backfills Break).
Object versioning turns an accidental overwrite into a restore rather than a loss, and it is one of the few genuinely cheap safety mechanisms available at this layer (Object Storage as Data Infrastructure).
Because arrival partitions are immutable once their interval passes, they are ideal candidates for lifecycle transitions to colder storage — the recovery position is preserved and the cost of holding it falls (Storage Lifecycle).

What can go wrong

Failure modes

In-place cleaning. Somebody fixes a bad partition by rewriting it, and the layer stops being evidence — not just for that partition, but for all of them, because nobody can now tell which were touched.
Analysts querying raw directly, producing dashboards that break every time a source changes shape, and generating pressure to make raw stable — which would require changing it (Data Platform Anti-Patterns).
Partitioning by event time, so late arrivals require rewriting old partitions and the immutability guarantee is lost by design rather than by accident.
Small files at short intervals, degrading every downstream read for the lifetime of the data (File Compaction).
The mitigation fails: retention configured to delete raw after ninety days while the platform advertises full reprocessability, so the guarantee is real for one quarter and false thereafter — and nobody notices until a backfill request crosses the boundary.
A source landing into a shared path with another source, so a re-run of one deletes the other's data. Path convention is a correctness mechanism, not tidiness.

Misreads

"Raw means messy." Raw means unmodified. It should be rigorously organised — consistent paths, arrival partitions, self-describing files, complete metadata. The content is untouched; the container is engineered (The Data Lake).
"We can clean it a bit on the way in." There is no bit. The first cast, filter or rename converts the layer from evidence into another derived table, and the value it provided disappears at that moment rather than gradually.
"Storage is expensive, so drop unused columns." Storage is the cheapest driver in a data platform and reprocessing a column you no longer have is impossible at any price. If a column must be dropped, drop it because it is classified, not because it is large (What Actually Drives Data Platform Cost).
"Raw is the source of truth." Raw is the source of *evidence*. The source system is the source of truth; raw is a faithful record of what it sent, which is a different and more useful claim during an incident (Source of Truth).
"The lake is the raw zone." A lake is object storage holding many layers, of which raw is one. Conflating them is how a lake becomes a swamp: a single bucket where landed, cleaned and modelled data are indistinguishable (The Data Lake).
"We keep raw, so we can always reprocess." Only if the transformations are deterministic. A model referencing now(), a mutable dimension, or a non-idempotent merge does not reproduce its own output from the same raw input (Idempotent Data Pipelines).

Privacy, retention and access

Landing everything means landing personal data with no stated purpose, which is precisely what data-minimisation principles exist to prevent. Resolve it by classifying at the boundary and excluding specific fields deliberately, so the exclusions are a reviewed list rather than each connector author's judgement (Data Minimization).
Deletion requests must reach raw, and raw is the hardest place to honour them: no primary key index, no update-in-place, and immutability as a stated design goal. The mechanisms are partition rewriting under a documented exception, crypto-shredding by discarding per-subject keys, or a retention window short enough that the request is satisfied by waiting (Deletion Requests).
Access to raw should be narrower than access to modelled data, not wider, because raw contains everything including fields that governance would have masked downstream (Data Masking, Tokenisation & Encryption).
Immutability and the right to erasure are in genuine tension. Naming the exception — how a partition may be rewritten, by whom, with what record — is better than an immutability rule that quietly does not apply (Data Retention).

Operating it

How you see it in production

Partition count, file count and average file size per source per day. The last of these is the leading indicator of a small-file problem, and it is invisible in every other metric (Scan Cost).
Bytes retained per source over time, split by age band. It is the input to the retention conversation and it makes the cost of the recovery position explicit rather than assumed (Storage Lifecycle).
A count of writes to partitions whose interval has already closed. Under a correct design this is zero except during declared recovery, and any other non-zero value is an immutability violation in progress.
Access logs on the raw location, specifically reads from principals that are not the staging pipeline. The first dashboard built on raw is much cheaper to prevent than to migrate (Audit Logs for Privileged Actions).

What changes at 10x and 100x

At 10x volume the partition granularity has to be revisited: hourly partitions that were reasonable become large, or daily ones become too coarse to reprocess selectively (Partition Cardinality).
At 100x, listing a prefix becomes an expensive operation in its own right and a table-format metadata layer starts earning its complexity — file lists are held in metadata rather than discovered by listing (Open Table Formats).
Source count scales the path-convention problem. With five sources, conventions are remembered; with two hundred, a convention that is not enforced programmatically is a convention that is not followed.
What does not change with scale is the rule. Land what arrived, do not modify it, partition by arrival. It is as correct for a single daily CSV as for a high-volume event stream, which is unusual in this domain and worth noticing.

What drives cost here

Bytes retained, indefinitely, including fields nobody uses. This is the honest cost of the layer and it is the cheapest resource in the platform per unit — but it accumulates monotonically because nothing ever deletes.
File count, which costs listing time and per-file read overhead on every downstream scan, forever. Driven by ingestion cadence and partition granularity together (File Size and the Small-Files Problem).
A second full copy of the data, since raw and the modelled layers coexist. That duplication is the price of reprocessability and should be stated as such rather than treated as waste (Raw, Staging, Curated: Layers by Purpose).
Lifecycle transitions reduce the retained-bytes cost substantially for old partitions, at the cost of retrieval latency when a deep backfill needs them — an excellent trade for a recovery position that is rarely read (Storage Lifecycle).

What this approach costs

Immutability buys total reprocessability and costs the ability to fix anything in place — including obvious mistakes, which is genuinely frustrating and is the whole point.
Keeping unused fields buys answers to questions not yet asked and costs storage plus a governance surface containing personal data with no current purpose. This is a real tension and the resolution is deliberate exclusion of classified fields, not a general policy of keeping less (Data Minimization).
Arrival partitioning buys bounded re-runs and immutable partitions, and costs read efficiency for event-time queries — which must scan several arrival partitions to assemble one event day.
A separate raw layer buys a recovery position and costs a second copy, a boundary to enforce, and a layer that consumers must be actively prevented from using.
Long retention buys deep recovery and costs a growing, permanent privacy liability. Neither argument wins outright and the answer is a documented retention period rather than a default (Data Retention).

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALLand unmodified, partition by arrival, never rewrite in place. This holds for object storage, for a warehouse raw schema and for a filesystem, and it is one of very few rules in this domain with no meaningful exception at any scale.
FORMAT-SPECIFICA self-describing format such as Parquet or Avro records the schema with the data, so a partition written under an older schema stays readable and a truncated file fails loudly on its missing footer. Headerless CSV does neither: a partial file parses cleanly, and the schema of an old partition exists only in whatever code wrote it.
ORG-SPECIFICThe retention period is a negotiation between recovery depth and privacy exposure, and the answer differs by jurisdiction, by data classification and by how long the organisation actually reprocesses. A platform that has never backfilled beyond a month is paying for a recovery position it does not use.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasehow-data-is-stored crash-recovery

Backendobject-storage presigned-urls file-processing

Securitydata-classification audit-logs encryption-rest-transit broken-access-control

Observabilityscan-vs-index-performance

Cloudobject-storage object-storage-architecture storage-lifecycle backup-strategy audit-trails

DSAhash-set

Domains that do not exist yet

— DevOps / Production Engineering owns the infrastructure-as-code that enforces the path convention, the bucket policy that makes raw genuinely read-only, and the lifecycle rules that move old partitions to colder storage.
— Distributed Systems owns what "immutable" means when an object store is eventually consistent for listings, and why a manifest is a more reliable statement of what a partition contains than a listing of it.