Schema Registry
A shared, versioned store of schemas with a compatibility gate in front of it. It makes structural evolution mechanical — and it has nothing to say about meaning, ownership or the consumers reading your data by some other path.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Where does a producer's schema live so that a consumer can resolve it, and what can that place actually refuse?
Two very different readers. A streaming consumer needs to resolve the exact writer schema for a record it is deserialising right now, in the hot path, cheaply. A human needs to know which versions of this dataset have existed, when each appeared, and what changed — which is the same store used as a changelog rather than as a lookup (Dataset Documentation).
The unit is one subject at one version. A subject is a named evolution lineage — usually one per topic or per record type — and versions within it are what compatibility is evaluated across. Two subjects have no relationship at all, which is why choosing what a subject corresponds to decides what gets checked.
Ship the schema inside every record. Every message carries its own field names, so any consumer can parse anything and no shared infrastructure is needed. JSON does exactly this and it is the reason JSON is everywhere (CSV, JSON and Their Limits).
The field names are re-sent on every record. At high volume the schema is a large fraction of the payload, and it is the same bytes every time (Dictionary, Run-Length, Delta and Bit Packing).
- The field names are re-sent on every record. At high volume the schema is a large fraction of the payload, and it is the same bytes every time (Dictionary, Run-Length, Delta and Bit Packing).
- Nothing is checked. A producer can change a field's type between one message and the next, and the first anyone hears about it is a consumer error — or, more often, a consumer that does not error (Breaking Schema Changes).
- There is no version history. When a consumer asks "what did this look like in March", there is no store that can answer, only whatever records still exist.
- Every consumer implements its own tolerance rules, so the same change is absorbed differently in six places and the platform has six different opinions about what the data is.
- A field that is optional in practice and required in one consumer's expectations is a landmine that nothing can detect until it goes off (Nullability & Defaults).
What is actually happening
- A registry stores schemas under subjects, versions them, and assigns each registered schema an identifier. The producer registers, receives the identifier, and writes a small reference to it alongside the record rather than the whole schema — so the schema travels once and the reference travels on every message (Avro).
- The consumer sees the identifier, fetches the corresponding writer schema, caches it, and resolves it against its own reader schema. After the first record of a given schema, resolution is a cache hit (Unsafe Deserialization).
- The gate is at registration time, not at write time. When a producer tries to register a new version of a subject, the registry compares it against previous versions under the subject's configured compatibility mode and refuses the registration if it violates it (Backward Compatibility).
- Which schemas are compared depends entirely on the subject naming strategy — whether a subject corresponds to a topic, to a record type, or to a combination. A topic carrying several record types under a topic-based strategy has all of them evolving in one lineage, which is usually not what anyone intended (Topics and Partitions).
- The registry is in the write path as a dependency. Producers and consumers cache aggressively so a registry outage does not stop steady-state traffic, but a *new* schema cannot be registered and an *unseen* schema id cannot be resolved while it is down (Pipeline Reliability).
Producer, schema, registry, gate
The flow is short and worth committing to memory, because most confusion about registries comes from thinking the check happens at write time. It does not. The check happens when a schema is registered, which is typically at producer start-up or in a deployment step — before any record has been written under it.
That timing is the registry's best property. Every other enforcement point in this module catches a bad change after data exists; this one catches it before, when the only cost of stopping is a failed deploy. It is also the reason a registry cannot help you at all with a producer that refuses to use it.
Follow the identifier through the diagram. It is what makes the whole arrangement cheap: the schema crosses the wire once, the identifier crosses it on every record, and the consumer's cache means the registry sees roughly one request per unique schema per client rather than one per message.
Compatibility mode names, their transitive variants, subject naming strategies and the exact wire format of the schema reference are product-specific and have changed across versions. Verify the current documentation of whichever registry you run before configuring a subject, and in particular confirm which direction its BACKWARD and FORWARD modes enforce — they are the opposite way round from how the API world uses those words.
What is stored, and what a subject actually is
A subject is an evolution lineage. Compatibility is only ever evaluated *within* one, which means the decision about what a subject corresponds to is the decision about what gets checked against what — and it is usually made by accepting a default.
The layout below shows the shape of the store. Two things in it are load-bearing. The identifier is globally unique and immutable, because data written years ago still references it. The version number is per subject and is what compatibility is evaluated across; a subject holding two unrelated record types will evaluate them against each other and reject perfectly reasonable changes.
The stage table underneath traces one record through the whole arrangement, with the guarantee each stage makes. Read the last two rows carefully: once the data has been ingested into a warehouse table, the registry is no longer involved in anything, and every guarantee it provided stops at that boundary.
- 1Register schema
Producer submits its schema for the subject at start-up or during deploy.
guarantees Either the schema satisfies the subject's compatibility mode against previous versions, or the registration fails and no data is written under it.
fails by Comparing against the immediately previous version only, so a slow consumer several versions back is not considered.
- 2Serialise and write
Encodes the record without field names and prefixes the schema identifier.
guarantees The record can always be interpreted later, because the identifier resolves to an immutable schema.
fails by A producer that bypasses the serialiser and writes raw bytes, which the registry cannot see and cannot refuse.
- 3Broker retains
Stores the record durably and lets many consumers read it independently.
guarantees Durability and replay within retention; per-partition ordering. Nothing about schemas (Retention and Replay).
fails by Retention expiring, at which point the schema is still resolvable and the data is not.
- 4Consumer resolves
Reads the identifier, fetches and caches the writer schema, resolves it against its own reader schema.
guarantees A deterministic resolution outcome per field: read, skipped, or filled from the reader's default (Forward Compatibility).
fails by Resolving successfully into a field the consumer then interprets under an assumption that changed.
- 5Ingest to warehouse
Lands the decoded records into a table with its own schema.
guarantees Only what the ingestion job asserts. The registry's guarantees end here.
fails by The table's schema drifting from the subject's, so two shapes describe the same data and neither is authoritative (Model Layering).
- 6Transform and serve
Models the table into facts and dimensions for query.
guarantees The grain and columns the model declares — a separate contract with a separate enforcement point.
fails by Assuming registry compatibility covers this layer, which it never did.
- 7BI consumer
Queries the serving table.
guarantees Nothing. This consumer has never heard of the registry.
fails by Being counted as protected in a coverage discussion about schema enforcement.
Five of the seven stages are downstream of the registry's last guarantee. That ratio is the honest summary of what a registry covers in a typical analytical platform.
registry
├── subject: orders.placed-value <- one evolution lineage
│ ├── version 1 -> schema id 41 <- immutable; still referenced by 2024 data
│ ├── version 2 -> schema id 57 <- added shipping_method (nullable, default null)
│ └── version 3 -> schema id 88 <- widened amount_minor to 64-bit
│ compatibility mode: <set per subject; check which direction it means>
├── subject: orders.cancelled-value
│ └── version 1 -> schema id 42
└── subject: payments.captured-value
├── version 1 -> schema id 43
└── version 2 -> schema id 91
record on the wire: [ schema id ][ ...encoded fields... ]
^ a few bytes, every message
^ no field names at allWhat it can refuse, and what it cannot see
A registry is the strongest structural enforcement point available and it is narrow. The table below is the honest boundary, and the misses column is the part worth memorising, because a team that believes a registry covers more than it does will stop looking for the failures it does not cover.
One pattern deserves emphasis. Everything a registry checks is decidable from two schemas. Everything expensive in this module — an enum growing inside a string field, a unit changing, a measure moving from gross to net, a producer quietly narrowing what it includes — is decidable from neither the schema nor the data. No configuration of any registry will ever reach those (Semantic Changes).
The correct posture is layered: the registry catches structural breaks before the data exists, a boundary check catches conformance violations as data arrives, data tests catch value-level anomalies after it lands, and a documented, owned, changelogged contract is the only thing standing between you and a meaning change (Contract Enforcement).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Compatibility gate at registration | A new schema version can be resolved against previous ones in the configured direction. | Removals, renames, type changes and nullability changes — before a single record exists under the new shape. | Every change with no schema difference; every producer that does not register; every consumer reading by another path. |
| Serialiser-enforced write | What was written actually matches the schema that was registered. | A producer whose code drifted from its declared schema, caught at the moment of writing. | A field that is present, correctly typed and carries a wrong value. Structure is not content (The Dimensions of Data Quality). |
| Allowed-value test at the ingestion boundary | A field the schema types as a string is in fact drawn from a closed set. | An enum gaining a value, which no structural comparison can see because the schema did not change (Enum Evolution: The New Value That Broke Old Clients). | A value inside the allowed set that is used to mean something new — the set is the same and the semantics are not. |
| Consumer-declared reader schema in the consumer's own repository | This consumer states what it depends on, so a compatibility check has something concrete to check against. | A change that breaks a specific consumer, at the producer's build time rather than at the consumer's run time (Contract Tests Between Services). | Consumers who never declared anything, which in most platforms is the majority — dashboards, notebooks and ad-hoc queries do not have repositories. |
| Documented semantics with an owner and a changelog | What each field means, who is accountable for it, and what changed when. | The meaning changes nothing else in this table can catch — if, and only if, a human reads it. | Anything the producer did not think to write down, and everything that happens while it is out of date. This is the weakest check here and it is the only one that covers the worst failure (Semantic Changes). |
Read this table as a stack rather than a menu. Each row catches something the row above it cannot, and the bottom row — the only one that is not automatable — is the one that covers the failures with the largest blast radius.
How to build it
Most important first.
- Set the compatibility mode per subject from the writer-and-reader sentence, record the intent in a comment or in the contract, and use the transitive variant where consumers migrate slowly — a chain of individually acceptable changes can otherwise strand a consumer several versions back. The mode names are the single most common configuration error in this area (Backward Compatibility).
- Choose the subject naming strategy deliberately, before there are subjects. Changing it later re-partitions the entire evolution history of every affected topic.
- Treat the registry as the changelog and give consumers a way to subscribe to it. A version history nobody is notified about is an audit trail, not a contract (Data Contracts).
- Keep the semantic clauses out of the registry and next to it — in the contract, in the catalog, in column descriptions. A registry stores structure; a description field is where meaning has to live (Semantic Changes).
- Enforce at a second point as well. A registry protects the path that goes through it, and much of a data platform reads files or warehouse tables by a path that does not (Contract Enforcement).
- Cache schemas in producers and consumers and treat registry availability as a dependency with its own monitoring, because a hard dependency in a write path is a reliability decision whether or not it was made deliberately.
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A registry guarantees that a schema referenced by an identifier can be retrieved and is immutable. That is a strong and genuinely useful guarantee: the schema a record was written with is knowable forever.
- It guarantees that registered versions of a subject satisfy the configured compatibility relation with each other. It does not guarantee the data matches the schema — that depends on the serialiser actually validating, and on the producer using it.
- It guarantees nothing about a consumer that reads the data through a different door: a warehouse table loaded from those events, a file export, a replica. Those readers never consult the registry (Data Engineering and Backend Engineering).
- It guarantees nothing about meaning, ownership, freshness, or whether a change was announced. Those are contract clauses and the registry has no field for them.
- It does not guarantee that a compatible change is a safe change. Compatible means resolvable; safe means the consumer's answer is still right (Semantic Changes).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check the registry itself performs is a compatibility check between schema versions at registration. It is cheap, deterministic and runs before any data exists, which makes it the earliest possible detection point in the whole module (Contract Enforcement).
- It misses every consumer expectation that is not expressed in a schema: allowed values in a string field, units, cross-field invariants, and the entire category of meaning.
- It also misses producers that do not go through it. A registry is only an enforcement point for the paths that use its serialiser, and a platform with one team writing raw JSON to the same topic has a gap the registry cannot see (Data Quality).
- In steady state the registry adds nothing to latency, because both sides cache. The cost is at cold start, on the first record of an unseen schema.
- A registry outage degrades in a specific and mostly benign shape: existing schemas keep resolving from cache, new registrations fail, and consumers that meet a genuinely new schema id block. That is a much better failure shape than most write-path dependencies have.
- A rejected registration stops the producer shipping, which is a freshness incident for every consumer of that dataset. The rejection must therefore reach a human quickly rather than being logged and retried forever (Quality Alerting).
- The registry is the mechanism by which schemas evolve, so its own evolution question is about policy: tightening a subject's compatibility mode retroactively invalidates nothing already registered but constrains everything after it, and that is a breaking change for the producing team (Schema Evolution).
- Subjects accumulate versions indefinitely. That is a feature — the point is that a record written two years ago can still be resolved — and it means the registry is a long-lived stateful service whose backup and restore matter more than its throughput.
- Migrating between registries, or changing subject naming, has to preserve identifiers, because those identifiers are embedded in data already written and will be dereferenced years from now.
- Losing the registry means losing the ability to interpret data that referenced it by identifier. Its store is therefore in the same reliability class as the data itself, not in the class of a stateless service (Pipeline Reliability).
- Recovering from a bad registration is a new version that restores the old shape, plus a decision about the data written in between — which is still on disk and still interpretable, because its schema is still in the registry.
- If a producer bypassed the registry and wrote data under a shape nobody registered, that data is recoverable only by inspection. This is the argument for the serialiser being the only way to write (The Raw Landing Zone).
What can go wrong
- The compatibility mode set from the name rather than from the intended direction, enforcing the opposite policy while looking correct (Backward Compatibility).
- A subject naming strategy that puts unrelated record types into one evolution lineage, so unrelated changes reject each other.
- Compatibility checked non-transitively, allowing a slow chain of changes to strand a consumer that has not upgraded in a year — or a producer path that bypasses the serialiser entirely, so the registry enforces a policy over a subset of the data nobody has measured.
- A registry outage during a deploy blocking a new schema registration and, with it, a release that had nothing else wrong with it.
- The registry treated as the whole of contracts, so meaning, ownership and freshness are nowhere (Data Contracts).
- Consumers reading the same data from the warehouse rather than from the stream, entirely outside the registry's protection (Data Engineering and Backend Engineering).
- "The registry is our data contract." It is the structural clause of one, mechanically enforced. Ownership, semantics, freshness and the change process are not in it (Data Contracts).
- "Compatibility passed, so consumers are safe." Compatibility means the bytes resolve. A consumer that resolves the record perfectly and computes a definition that changed last week is not safe (Semantic Changes).
- "A registry is a Kafka thing." The pattern — a versioned schema store with a compatibility gate — is independent of the transport, and the same idea appears as table-format schema metadata, as contract files in a repository, and as column definitions in a catalog (Open Table Formats).
- "Schemas in the registry describe our warehouse tables." They describe what a producer wrote to a topic. The warehouse table is downstream of a transformation that may have renamed, cast, filtered and joined, and its shape is a separate contract (Model Layering).
Operating it
- Registrations and rejections per subject over time. A spike in rejections is a team trying to make a change the policy forbids, and it predicts a workaround (Contract Enforcement).
- Schema identifiers observed in the last window per topic, which is how you discover producers you did not know were writing.
- Consumer-side schema resolution errors and cache misses, which distinguish "the registry is slow" from "someone shipped a new schema".
- Version count and age per subject, as the raw material for a changelog consumers can actually read (Metadata: Technical, Operational and Business).
- At 10x producers, the registry's value shifts from byte savings to policy: it is the only place where "what may change" is enforced once rather than argued per team.
- At 10x throughput, nothing changes in the registry — resolution is cached and the registry sees roughly one request per unique schema per client, not per record.
- At 100x subjects, the human problem dominates: a registry with thousands of subjects and no ownership metadata is a schema landfill, and it needs the catalog next to it to be navigable (The Data Catalog).
- The registry replaces per-record schema bytes with a small per-record reference, so the saving grows with record count and with how verbose the schema is relative to the payload (Dictionary, Run-Length, Delta and Bit Packing).
- Its own operational cost is small and constant — a low-traffic service holding a small amount of data — and its reliability requirement is high, which is an unusual combination that surprises teams sizing it.
- The cost people underestimate is coordination: a gate that refuses changes converts some engineering work into conversations, which is the point and is not free (Data Contracts).
- A registry adds a stateful dependency to the write path in exchange for smaller records and an enforcement point. Caching makes the dependency mild, and mild is not zero.
- It enforces structure precisely, which makes it tempting to treat structural compatibility as the definition of safety. That substitution is the most expensive mistake this lesson can cause (Semantic Changes).
- It protects one path. In a platform where most consumers read a warehouse table rather than the stream, most consumers are outside it, and the registry can produce a confident feeling of coverage that the data does not support.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe pattern — subjects, versions, immutable identifiers, and a compatibility gate at registration — is transport-independent and has been stable for a decade. What varies is where it physically lives: a standalone service for streams, table metadata in an open table format, or contract files checked in a producer's repository.
- BROKER-SPECIFICThe registry pattern is most developed around Kafka, where the record carries a schema reference in its bytes; other transports handle this differently or not at all, and a system that passes plain JSON over HTTP has no equivalent hook at which a schema reference could ride along.
- TOOL-SPECIFICCompatibility mode names, subject naming strategies and the wire format for the schema reference are product decisions, not properties of the idea. Two registries can implement the same concept and disagree on which changes their similarly-named mode permits.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the availability question a registry raises: a small, strongly consistent, highly available store in the write path of everything, and what a partition between a producer and that store actually does.
- — DevOps / Production Engineering owns registering schemas as a deployment step, and what a rollback means when the previous release registered a version the current one cannot produce.