ContractsGENERALORG-SPECIFICSOURCE-SPECIFIC

Data Contracts

An explicit, owned, enforced agreement between the team that produces data and the teams that depend on it — covering names, types, nullability, meaning, freshness and how it may change.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

What has the producing team actually promised about this dataset, and who finds out first when the promise stops being true?

Who needs this

Every team that reads the dataset without being able to read the producer's code: analysts writing SQL against it, transformation models built on it, a feature pipeline that trains on it, a finance close that signs off numbers derived from it. What they need is not the schema — they can see the schema — but the parts they cannot see: what each field means, whether null is possible, how fresh the table is, and what will happen to them the next time the producer ships a migration.

What one row is

The unit of a contract is one field of one dataset or event type. A contract that names only the dataset — "we publish orders" — is a title. The commitments that matter are per field: amount_minor is an integer, never null, in minor units of the currency field, gross of refunds, and it will not change type without a version bump. Everything painful in this module happens at field granularity.

The obvious build

Write the schema down. Someone exports the DDL, or a JSON Schema, or an Avro file, into a wiki page or a schemas/ directory, and calls it the contract. It is a real improvement on nothing at all — consumers now have somewhere to look — and for a while nobody notices what it is missing.

Why it breaks

A migration renames amount_cents to amount_minor. The schema file is updated in the same pull request, honestly and correctly. No consumer is notified, because nothing in the system knows who the consumers are, and every revenue model that referenced the old column produces null (Breaking Schema Changes).

How it breaks with real data

A migration renames amount_cents to amount_minor. The schema file is updated in the same pull request, honestly and correctly. No consumer is notified, because nothing in the system knows who the consumers are, and every revenue model that referenced the old column produces null (Breaking Schema Changes).
A field documented as "not null" is not enforced anywhere, so the first null arrives eighteen months later from a code path that did not exist when the document was written. The downstream AVG() silently changes its denominator (Nullability & Defaults).
The contract says status is one of five values. A sixth appears. Every consumer that wrote a CASE statement over the five now maps the sixth to its ELSE branch, which is usually the least conservative bucket (Data Quality).
Nobody can say who owns it. The team that wrote the service was reorganised, the schema file has three authors, and the incident channel spends the first hour of every data outage determining whose problem it is (Data Ownership).
revenue changes from gross to net. The schema file is still correct, because nothing in it was ever about meaning. Every dashboard is now wrong and no check anywhere will fire (Semantic Changes).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

A contract is the interface boundary of a dataset, in exactly the sense that an API signature is the interface boundary of a service (What an API Contract Actually Is). Without one, consumers do not depend on an interface — they depend on the *current implementation* of a producer's storage, which the producer is entitled to change at any time and usually does.
What makes it a contract rather than a description is that it has three properties a description does not: an owner who can be asked, an enforcement point where a violation is detected before it reaches a consumer, and a change process that says what may change silently, what requires notice, and what requires a version.
The content is broader than a schema. Field names and types are the easy half. The other half is nullability, allowed values, units, the grain of one row, the semantics of each measure, the freshness the producer commits to, the delivery semantics of the transport carrying it, and the compatibility policy under which it may evolve (Schema Evolution).
Contracts move a cost rather than removing it. Without a contract the cost lands on consumers, later, as incidents, and it is paid many times over by whoever happens to be on call. With a contract it lands on the producer, earlier, as a rejected change or a required version bump. The total work is not obviously lower; the distribution is fairer and the failures are louder (Contract Enforcement).
The mechanism that makes contracts stick is consumer visibility. A producer who can list the datasets and dashboards that depend on their table behaves differently from one who cannot, and that list comes from lineage rather than from goodwill (Impact Analysis).

What is actually in a contract

SIMPLIFIEDReal contract formats add ownership metadata, classification, SLAs and links into a catalog, and are usually generated rather than hand-written. The clause list is the stable part; the serialisation is not, and the choice between JSON Schema, Avro, Protobuf and a bespoke YAML matters far less than whether anything reads it.

Most teams that say they have data contracts have a schema file. The gap between those two things is where every contract-related incident lives, so it is worth being concrete about the clauses a schema file does not contain.

Read the shape below clause by clause and ask, for each one, what would detect a violation. Types and required fields are decidable from a single record. Allowed values are decidable from a single record. Units and semantics are not decidable at all, which is exactly why they must be written down for a human — they are the clauses no validator will ever cover for you. Freshness is decidable only by watching arrivals over time, and only from the consumer side.

The last three keys are the ones that turn the file into a contract. owner says who can be asked and who is accountable. compatibility says what may change without a conversation. enforced_at says where a violation is caught, which is the difference between a promise and a mechanism.

The shape of a contract — not any product's format, but the clauses that have to exist somewhere

1dataset: orders.placed
2version: 3
3owner:
4  team: checkout
5  oncall: checkout-oncall
6grain: one placed order, one row, keyed by order_id
7freshness:
8  commitment: complete for a given hour by the end of the following hour
9  measured_by: consumer-side arrival lag on the raw landing zone
10delivery: at-least-once; consumers must deduplicate on event_id
11 
12fields:
13  - name: order_id
14    type: string
15    nullable: false
16    semantics: the checkout service's order identifier; stable forever
17  - name: placed_at
18    type: timestamp
19    nullable: false
20    semantics: commit time in UTC, not the client's local clock
21  - name: amount_minor
22    type: integer
23    nullable: false
24    semantics: >
25      gross of refunds and gross of tax, in minor units of the currency field,
26      at the price agreed when the order was placed. NOT net.
27  - name: currency
28    type: string
29    nullable: false
30    allowed: [EUR, USD, GBP]
31  - name: status
32    type: string
33    nullable: false
34    allowed: [placed, cancelled]
35    semantics: closed set; a new value requires a version bump
36 
37compatibility:
38  may_ship_freely: [add a nullable field, widen an allowed-value set on a field no consumer branches on]
39  requires_notice: [remove a field, narrow an allowed-value set, change nullability]
40  requires_new_version: [rename a field, change a field's type, change a field's meaning]
41 
42enforced_at:
43  - producer CI: contract test against the emitted payload
44  - ingestion boundary: reject the batch on a schema or allowed-value violation

The clause doing the most work is semantics on amount_minor. It is the only clause that would have prevented a change from gross to net, and it is the only one no tool can check.

A contract without an owner and an enforcement point is a document

The two diagrams below are the same pipeline. In the first, the contract is a file that a human reads when they remember to. A schema change ships, the data flows, and the contract is not in the path of anything — it is beside the pipeline, not inside it.

In the second, the same file is loaded by two checks: one in the producer's continuous integration, which fails the build when the emitted payload no longer matches, and one at the ingestion boundary, which refuses a non-conforming batch. Now a violation has a detection point and an owner to route to. Note what this costs: the second design has a way to fail that the first does not have, and it will use it.

The producer-side check is the valuable one, because it catches the change before the data exists. The ingestion-side check is the necessary one, because you do not control every producer — and for a third-party source it is the only check you will ever have (Ingestion Sources).

The same contract, beside the pipeline and inside it

UserLLMAgentToolDataDecisionHumanGuardrail

The contract is with a consumer, not with a schema file

The failure mode that survives even a well-enforced contract is a consumer depending on something the contract never mentioned. A column that exists but was never promised, an ordering that happens to hold, a value that has always been present in practice — all of these become dependencies the moment someone writes SQL against them, and none of them is protected.

This is the same problem as an internal service accidentally exposing its storage model to callers, and the same fix applies: publish a deliberately shaped view rather than the raw table, so what consumers can depend on and what you have promised are the same set (Three Models, Not One).

It also changes what a contract negotiation looks like. The useful question to a consumer is never "is this schema acceptable" — they will say yes — but "which fields will you branch on, which will you aggregate, and what will you do when one is null". The answers tell you which clauses actually have to be enforced and which are documentation.

Names and types — the part every tool already checks, and the part that causes the fewest surprises.
Nullability — enforced far less often than it is documented, and the source of a whole class of quiet aggregate drift (Nullability & Defaults).
Allowed values — a closed enum is a promise that a sixth value will not appear without notice; almost nobody enforces it and almost every consumer assumes it.
Units and semantics — minor units or major, gross or net, UTC or local, inclusive or exclusive. Unenforceable and indispensable (Semantic Changes).
Grain — what one row is. A contract that does not state it invites every double-counting bug in the modelling module (Grain: What Does One Row Represent?).
Freshness and delivery — how late data may be and whether it may arrive twice (The Freshness SLO).
Owner and change policy — who to ask, and what may change without asking (Data Ownership).

Two ways of publishing the same orders data

Expose the producer's table

Consumers read `checkout.orders` directly — every column the service happens to have, including internal state flags, denormalised caches and the columns added last sprint for a feature that was reverted. The contract, if it exists, describes the six columns anyone thought to write down.

Publish a shaped, versioned dataset

The producer publishes `orders.placed@v3` containing exactly the fields it is willing to commit to, derived from its internal tables. Internal columns are not present. A consumer cannot accidentally depend on something unpromised, because it is not there to depend on.

You cannot enforce a contract over fields nobody agreed to, and you cannot rename an internal column safely once an analyst has written a dashboard on it. Publishing a narrower surface makes the promised set and the depended-on set identical, which is the only state in which "this change is safe" is a statement anyone can verify rather than hope.

How to build it

Most important first.

Write the contract from the consumer side first. Start from the questions consumers must be able to answer and the guarantees they need in order to answer them, then ask the producer which of those they are willing to commit to. A contract drafted producer-first tends to codify whatever the source table happens to look like today (Consumer-First Design).
Put the semantics of every measure in the contract, in prose, next to its type. revenue_minor: integer is not a commitment; revenue_minor: integer — gross of refunds and taxes, in minor units of the currency field, at order-placement time is (Semantic Changes).
Name one owning team per contract, with an on-call route. Ownership by "the data team" for a field produced by a product service is ownership by nobody, because the data team cannot make the field correct (Who Owns Data Quality).
Declare the compatibility policy explicitly — which changes may ship freely, which require a deprecation window, which require a new version — and state it in terms of who reads what, not in terms of the words "backward" and "forward", which two communities use in opposite directions (Backward Compatibility).
Give the contract a machine-readable form, one place where it is checked, and a home in the producer's repository next to the code that satisfies it — so a change to the shape and a change to the promise are the same pull request. A contract that exists only in prose degrades into folklore within two quarters (Contract Enforcement).
Include a freshness commitment and a stated delivery semantic. Consumers routinely assume both, and their assumptions are always more generous than the truth (The Freshness SLO).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

A contract guarantees exactly what its enforcement point can detect and refuse. An unenforced clause is a wish with formatting.
A schema clause can be enforced completely: types, presence, allowed values and ranges are decidable from the payload alone (Transport Validation).
A semantic clause cannot be enforced at all by machine. "Gross of refunds" is not checkable against a single record; at best a distribution test notices that the number moved (Distribution Tests).
A freshness clause is enforceable only downstream, by observing arrival — the producer can promise it and only the consumer can measure it (Freshness Monitoring).
Nothing in a contract guarantees the producer is right. A field that is correctly typed, non-null, in range and delivered on time can still describe something that did not happen.

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The check that gives a contract teeth is a conformance test at the boundary: validate every arriving batch or event against the declared schema — types, required fields, allowed values — and refuse it if it does not conform (Data Tests).
It misses everything a payload cannot reveal. A perfectly conformant record can hold a value in the wrong currency, an amount that is net where the contract says gross, or a timestamp in the wrong zone.
It also misses absence. Conformance is evaluated on what arrived; a producer that sends nothing sends nothing that violates the schema, which is why a completeness or freshness check must sit next to it (Freshness Checks).

Freshness

A contract itself adds no latency. What adds latency is enforcement placed in the path of the data, which turns a violating batch into a rejected batch and therefore into a gap rather than into wrong values (Contract Enforcement).
The freshness clause is the part consumers most often need and producers most often omit. It should state a shape — "hourly, complete for a given hour by the end of the following hour" — rather than a number pulled from what the pipeline happens to achieve on a good day.
A contract that promises freshness the producer does not measure is worse than one that promises nothing, because consumers will build decisions on it.

When the schema or meaning changes

The contract is the thing that evolves, and the whole rest of the module is about how. The two questions that matter are which changes may ship without coordination, and how a change that cannot ship that way is sequenced (Schema Evolution).
Semantic evolution is the hard case and it needs its own clause: a change in meaning must be treated as a breaking change even when the schema is untouched, and announced through the same channel (Semantic Changes).
A contract should carry a version and a changelog that consumers can subscribe to. Most organisations discover they need the changelog during their second incident, not their first.

How to re-run this safely

When a contract is violated and the violation was caught, recovery is a re-run: fix the producer, replay the rejected range, republish. That is only possible if the rejected data was retained rather than dropped (The Raw Landing Zone).
When a contract is violated and the violation was *not* caught, recovery is a backfill of every downstream dataset for the affected period, in dependency order, plus a correction to anyone who acted on the wrong numbers (Planning a Backfill).
The recovery position is set long before the incident, by whether raw arrivals are immutable. A platform that transforms on ingest has no recovery from a contract violation it did not detect (Keeping Raw History: The Recovery Position and the Liability).

What can go wrong

Failure modes

The contract exists and nothing checks it. This is the common case and it is worse than having none, because consumers trust it.
The contract is checked at a point that is already downstream of the damage — for instance in the warehouse, after a transformation has already cast bad values to null (Breaking Schema Changes).
The contract is enforced so strictly that the ingestion boundary rejects data for cosmetic reasons, teams learn that rejections are noise, and the enforcement is disabled during the next incident and never re-enabled (Alert Fatigue: The Page Nobody Reads).
The contract describes fields but not meaning, so it survives every schema check while the metric it feeds becomes wrong (Semantic Changes).
Ownership drifts after a reorganisation and the contract outlives the team that could honour it.
Consumers depend on fields the contract does not mention — undocumented columns that happened to be present — so a change the producer was entitled to make breaks someone anyway (Schema Leakage).

Misreads

"We have a schema, so we have a contract." A schema is one clause of a contract, and it is the clause that tooling already enforces. The clauses that carry the incidents — meaning, nullability in practice, freshness, ownership — are the ones nobody wrote down (Semantic Changes).
"The contract protects consumers." It protects consumers from *unannounced* change. It does nothing about a producer whose data is wrong in a conformant way, which is most of what a data quality programme deals with (The Dimensions of Data Quality).
"Contracts are a governance concern." They are an engineering interface. Governance cares about who may see a field; a contract cares about whether the field still means what it meant last week (Data Governance).
"If the producer signs it, we are safe." A signature without an enforcement point is a document. The question to ask about any contract is where the check runs and what happens to the batch when it fails (Contract Enforcement).

Privacy, retention and access

A contract is the natural place to record a field's classification, because the producer is the only party who knows whether a new column contains personal data (Data Classification).
Adding a field to a contract can add a privacy obligation to every downstream consumer at once. A contract review is therefore a reasonable place to ask whether the field should be published at all (Data Minimization).
Retention belongs in the contract too: consumers who build on a dataset need to know how far back it will still exist when they backfill (Data Retention).

Operating it

How you see it in production

Contract violations per producer per day, as a first-class metric. A producer with a rising violation rate is a schema change in progress that nobody announced.
The list of consumers per dataset, derived from query logs and lineage rather than from a registry someone maintains by hand (Data Lineage).
Contract version in the metadata of the produced data, so a consumer reading a partition can tell which promise it was written under (Metadata: Technical, Operational and Business).
Time from a producer merging a schema change to the first consumer noticing. In an organisation without contracts this number is measured in incidents rather than in minutes.

What changes at 10x and 100x

At ten datasets, contracts are ceremony and a shared channel works better. The mechanism only pays for itself once no single person can hold the dependency graph in their head (The Data Catalog).
At 10x consumers, the value of the contract shifts from schema stability to blast-radius knowledge — the question stops being "will this break" and becomes "who exactly does this break" (Impact Analysis).
At 100x datasets, per-dataset contracts written by hand stop being maintainable and the contract has to be generated from the producer's code and checked in the producer's CI, or it will not exist (The Self-Service Data Platform).

What drives cost here

The direct cost is validation compute at the boundary, proportional to records validated and to how deeply each record is inspected. Structural validation is cheap; full value-level validation of every field on every record is not (Scan Cost).
The real cost is coordination. A contract makes some changes require a conversation, and conversations across teams are the most expensive resource a platform consumes.
The cost avoided is incident-shaped and therefore invisible on any dashboard: the backfills that did not happen, the quarters that were not restated, the trust that was not lost.

What this approach costs

Contracts slow producers down. That is the mechanism, not a side effect: the whole point is that a change which used to cost the producer nothing and cost consumers an incident now costs the producer a version bump. Teams that will not accept that slowdown do not have contracts, whatever their wiki says.
Enforcement converts silent wrongness into loud failure. This is the correct trade and it will still page someone, and the person paged is often not the person who caused it (Contract Enforcement).
A contract is another artefact to keep true. A stale contract actively misleads, so the maintenance is not optional once you have started.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThe three properties that separate a contract from a document — an owner, an enforcement point and a change process — hold whatever the transport is. What changes is where the enforcement can physically sit: an event stream allows rejection per record, a nightly file drop usually only allows rejection per batch.
ORG-SPECIFICContracts solve a coordination problem between teams. With one team producing and consuming, they are overhead and a shared test suite does the same job; with a product organisation of thirty teams they are the only thing that keeps upstream changes from being discovered downstream as incidents.
SOURCE-SPECIFICA contract over an internal service you control can be enforced in the producer's CI before the data exists. A contract over a third-party SaaS export cannot: the vendor did not sign it, so the only available enforcement is rejection at your ingestion boundary, and the only available remedy is a support ticket.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databaserequirements-to-schema

Backendcontract-tests schema-leakage dtos-and-api-models

API Designwhat-is-an-api-contract consumer-driven-evolution consumer-first-design request-contracts response-contracts api-ownership

Observabilityalerting

Domains that do not exist yet

— DevOps / Production Engineering owns the delivery mechanics that make a contract enforceable in practice: running the conformance check in the producer's pipeline, gating the merge on it, and rolling back the producer when a violation reaches production.
— Distributed Systems owns the delivery semantics a contract's delivery clause refers to — what at-least-once actually promises across machines, and why a consumer must deduplicate rather than trust it.