Data Architecture Patterns
Central warehouse, event-driven platform, Lambda, Kappa and mesh — sorted onto the two independent axes they actually live on, and compared by the problem each was a response to.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Each of these patterns has a name, a diagram and an advocate. Which problem was each one solving, and which of those problems do I actually have?
The people who will live inside the answer: analysts who need one definition of revenue, domain teams who want to ship a dataset without filing a ticket, and the engineers who will be paged when whichever arrangement you chose degrades at three in the morning (Who Actually Consumes This Data).
The unit being compared is a platform posture — one organisation's standing answer to who produces data, who transforms it, who owns the result and how it moves. Not a product, not a diagram, and not a storage format. Comparing a posture with a storage choice is the single most common category error in this material (Lake vs Warehouse vs Lakehouse).
Pick the pattern that appears most often in recent conference talks and adopt it. This is more defensible than it sounds — these patterns encode real experience, and copying an arrangement that worked at a company you respect is a reasonable starting hypothesis when you have no other information.
The pattern was a response to a constraint you do not have. Lambda exists because the batch engine and the stream engine of its era could not be the same system; if yours can, you inherited the complexity without the reason (Lambda Architecture).
- The pattern was a response to a constraint you do not have. Lambda exists because the batch engine and the stream engine of its era could not be the same system; if yours can, you inherited the complexity without the reason (Lambda Architecture).
- Two patterns get argued as alternatives when they are on different axes. "Should we do mesh or lakehouse?" has no answer, because one is about who owns the dataset and the other is about who owns the table concept (Data Mesh).
- The pattern is adopted as a diagram rather than as an operating model. Boxes get built; the ownership, the governance forum and the on-call rota that made the pattern work do not, because they were never on the diagram (Data Platform Engineering).
- The organisational prerequisite is missing. Domain-owned data products require domain teams with data engineering capacity; without it the pattern names owners who cannot do the work and datasets degrade with an owner's name attached (Data Ownership).
- The decision is made once, at a moment when volume, source count, consumer mix and team size all looked different, and then defended by history. Every one of these patterns is right for some organisation and wrong for the same organisation five years later.
What is actually happening
- These patterns live on two independent axes, and almost every unresolvable architecture argument is two people standing on different ones. The first axis is dataflow: is the authoritative computation batch, streaming, or both at once. Lambda and Kappa are answers here, and so is a plain nightly warehouse (Batch vs Streaming Ingestion).
- The second axis is ownership: does one central team model and serve all data, or do the teams that produce the data also model and serve it. The central warehouse and the data mesh are answers here, and neither of them says anything about batch versus streaming (The Central Warehouse).
- A third axis exists and belongs to another module: storage posture — files, an analytical database, or a table format over files. It is orthogonal to both of the above, which is why every combination of the three is buildable and several are common (Lake vs Warehouse vs Lakehouse).
- The event-driven data platform is a dataflow pattern with an ownership consequence attached. Publishing to a shared log decouples producers from consumers, which makes domain ownership *possible* — which is why mesh discussions and event-log discussions keep colliding (The Event-Driven Data Platform).
- Every pattern here trades coordination cost against consistency of meaning. Centralising modelling gives one definition of a metric and one queue to wait in. Distributing it gives many queues of length zero and many definitions of
active customer. There is no arrangement that gives both, and the entire debate is about which failure your organisation survives better (The Metrics Layer).
Two axes, not one league table
The patterns in this module are usually presented as a list, which invites the question "which is best" and guarantees an unproductive answer. They are better read as points on two independent axes, because that immediately explains which pairs are genuine alternatives and which pairs are simply a combination somebody has already built.
The dataflow axis asks where the authoritative computation happens: in a batch job over accumulated data, in a continuous stream job, or in both simultaneously with a merge at the end. The ownership axis asks who models the data: a central team that serves everyone, or the domain teams that produce it. A nightly central warehouse sits at batch-plus-central. A mesh of domain teams publishing from a shared log sits at streaming-plus-distributed. Both are common; neither is a contradiction.
Once the axes are separated, several long-running arguments dissolve. Lambda versus mesh is not a question. Kappa versus lakehouse is not a question. What remains are the two real questions — how does data flow, and who owns the meaning — and each has a small number of honest answers with clearly different costs.
| Pattern | Axis it lives on | The problem it was a response to | What it genuinely buys | What it costs to run |
|---|---|---|---|---|
| Central warehouse | Ownership: centralised | Every team answering the same question differently, and analytics competing with production for capacity. | One place, one definition, one team that knows where everything came from. Governance and access control have a single enforcement point. | A queue. The central team's throughput becomes the organisation's analytics throughput, and its domain knowledge thins as sources multiply. |
| Event-driven platform | Dataflow: streaming, with an ownership consequence | Producers having to know, and change for, every consumer of their data. | Producers publish once; any number of consumers subscribe independently, replay history and materialise their own view. | Every consumer independently handles duplicates, ordering and schema evolution — one problem solved once becomes many problems solved unevenly. |
| Lambda | Dataflow: batch and streaming together | Batch results being trusted but hours old, while stream results were fresh but not trusted. | A fresh answer now and a correct answer later, from one serving interface, without asking consumers to choose. | Two implementations of the same business logic in two systems, which must agree forever and diverge silently when they do not. |
| Kappa | Dataflow: streaming only | The duplicated logic that Lambda requires. | One codebase, one engine, one mental model. Reprocessing is a replay of the same job rather than a second implementation. | The log must retain everything you might reprocess, and the stream job must be able to replay history at a rate the batch layer used to manage. |
| Data mesh | Ownership: distributed | A central team becoming a bottleneck and losing the domain knowledge needed to model a hundred sources correctly. | The team that knows what a field means owns the dataset that exposes it, and can ship without a ticket. | Every domain needs data engineering capacity, federated governance needs real authority, and cross-domain joins become cross-team negotiations. |
| Lake / warehouse / lakehouse | Storage posture — a third, orthogonal axis | Where the bytes live and who owns the table concept. | Covered in the storage module; it is compatible with every row above. | Not comparable with the rows above. Arguing it against them is the category error this table exists to prevent (Lake vs Warehouse vs Lakehouse). |
Which constraint do you actually have
A pattern is worth adopting when it removes a constraint you can name. If nobody can complete the sentence "the thing we cannot currently do is ___", then the honest recommendation is to change nothing and revisit when somebody can.
The four constraints below cover almost every real case, and each maps to a different axis. Notice that two of them are not solved by any pattern in this module: if the problem is that a query scans too much, the answer is in the layout module, and if the problem is that a metric is wrong, the answer is in modelling and quality. Reaching for an architecture pattern to solve either is a very expensive way to avoid a smaller piece of work.
Notice also what the options cost. Every row buys something and charges for it in coordination, in duplicated implementation, or in maintenance nobody put on a diagram. The criteria are the lesson here; there is deliberately no recommended leaf.
Name the thing your organisation currently cannot do. Which of these is it?
when The central team is the throughput limit, and the requests are mostly domain-specific rather than cross-cutting.
cost Moving along the ownership axis. It costs data engineering capacity in every domain team and a governance forum with real authority — and it is not available to you until a self-service platform exists (Data Mesh).
when A decision genuinely changes based on data younger than the current batch interval — an operational alert, a fraud response, a live inventory view.
cost Moving along the dataflow axis. It costs stream processing expertise, event-time reasoning and a second set of failure modes. Verify the decision really changes first; most dashboards demanding minutes are read once a day (Batch vs Streaming Ingestion).
when Every schema change upstream triggers a coordination exercise, and producers have started avoiding changes rather than making them.
cost Publishing to a shared log and putting a contract on the event. It costs each consumer its own handling of duplicates, ordering and evolution (The Event-Driven Data Platform).
when The same business concept is computed several ways and nobody arbitrates.
cost This is not an architecture problem. It is a metrics-definition and ownership problem, and both a central warehouse and a well-governed mesh can solve it — while neither solves it automatically (The Metrics Layer).
when The complaint is scan volume, runtime or bill rather than availability of data or speed of delivery.
cost Also not an architecture problem. Partitioning, clustering, file size and incremental processing move this by far more than any pattern here does (Partitioning, File Size and the Small-Files Problem).
when The platform meets its consumers' needs and no team is waiting.
cost None. Write down the triggers that would change the answer — a new consumer class, a source-count jump, a governance obligation — and revisit on those rather than on a calendar.
Patterns compose; they do not replace
Real platforms are almost never a pure instance of one pattern. The common shape in a large organisation is a central platform team providing shared capabilities, an event log carrying operational changes, several domain teams publishing products onto it, and a central warehouse still serving the finance models that must have exactly one definition. That is not indecision — it is four patterns applied where each is strongest.
The diagram below shows that composition. Read the ownership boundary rather than the arrows: the platform team owns the capabilities, the domain teams own the meaning of their products, and the governance forum owns the small set of definitions that must be identical everywhere. Every unhappy version of this diagram has that boundary in the wrong place or nowhere at all.
What makes a composition healthy is that each layer has a named owner and a stated promise. What makes it a swamp is arriving at the same picture by accretion, where the event log exists because one team wanted it, the warehouse exists because it always has, and nobody can say which of the two is authoritative for a given concept (Source of Truth).
Definition disputes, access requests, schema negotiations and waiting. The largest cost of any ownership arrangement and the one that appears on no bill, which is why it is routinely omitted from the comparison.
Structural in Lambda, emergent in event-driven platforms where several consumers derive the same thing, and near zero where one team models once.
Ingestion connectors, orchestration, catalog, test framework and observability. Fixed rather than proportional to data, and it rises sharply the moment domain teams are expected to self-serve.
Log-centric arrangements pay storage for exactly the replay horizon they promise; batch-centric ones pay it in raw and intermediate layers. Same driver, different place.
Almost always smaller than the four above and almost always the first thing anyone tries to optimise, because it is the one with a dashboard.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative and unitless, to establish an ordering rather than to predict anything. The teaching is the top row: architecture patterns are chosen and abandoned for reasons that live in the coordination line, and any comparison that only models infrastructure is comparing the smaller half.
How to build it
Most important first.
- Name the constraint before naming the pattern. Write one sentence: "the thing we cannot currently do is ___, and it is blocked by ___." If that sentence names a queue of analytics requests, you have an ownership problem; if it names data being hours old when a decision needs minutes, you have a dataflow problem. They have different answers.
- Place your candidate patterns on the two axes before comparing them. Two patterns on the same axis are genuine alternatives; two on different axes are a combination, and arguing them against each other wastes a quarter.
- Prefer the arrangement with the fewest moving parts that satisfies the constraint, and write down the trigger that would make you revisit (Data Platform Anti-Patterns).
- Whatever the arrangement, keep the raw arrival immutable and outside it. That single property is what makes every one of these decisions reversible, and it is the reason a wrong choice here is survivable (Keeping Raw History: The Recovery Position and the Liability).
- Separate the platform decision from the ownership decision explicitly. A central platform team providing shared capabilities is compatible with both centralised and distributed modelling, and conflating the two is why "we tried mesh" often means "we disbanded the platform team" (The Self-Service Data Platform).
- Write the operating model down next to the diagram: who owns each dataset, who is paged for it, who arbitrates a definition dispute, and what a new consumer has to do to get access. A pattern without those four answers is decoration (Data Governance).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- No pattern on this list guarantees that data is complete or correct. Completeness is a property of your ingestion and your reconciliation, and it is unchanged by whether the modelling is central or distributed (Reconciliation).
- The central warehouse guarantees one definition per metric inside its boundary, and guarantees nothing about the extracts, notebooks and shadow pipelines outside it — which is where its actual failure lives.
- The event-driven platform guarantees durable, replayable, per-partition-ordered delivery to any number of independent consumers. It explicitly does not guarantee that two consumers of the same topic computed the same thing (Topics and Partitions).
- Lambda guarantees eventual convergence of the batch view onto the truth, and guarantees nothing about the speed view agreeing with it in the meantime. Kappa guarantees that a replay is reproducible, provided the log still holds the input and the job is deterministic (Deterministic Replay: Making the Schedule Reproducible).
- Mesh guarantees nothing technically at all. It is an allocation of responsibility, and every guarantee under it is one a domain team wrote into a data product's contract (Data Contracts).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that transfers across all of these is a closed-period reconciliation against the source system: row count and a summed measure, compared between the system of record and whatever the consumer actually reads (Reconciliation).
- Under a distributed ownership model, add a second check that is specific to the arrangement: definition divergence. Compute the same headline metric from two domains' products and alert when they disagree by more than a stated tolerance. That failure cannot happen in a central warehouse and is the characteristic one in a mesh (Two Dashboards, Two Numbers).
- Both checks miss everything about open periods, everything that is wrong identically at both ends, and every column they do not sum. Neither can tell you a metric is *meaningful* — a reconciled
revenuethat silently switched from gross to net reconciles perfectly (Semantic Changes).
- Only the dataflow axis moves freshness. Choosing domain ownership over central ownership does not make anything fresher; it changes who is able to make it fresher without asking permission.
- Lambda's freshness is deliberately two-valued: the serving layer answers from a batch view that is as old as the last batch run, merged with a speed view covering the gap. Consumers who do not know that get an answer whose staleness varies by which half of it they hit (Freshness Monitoring).
- Kappa's freshness is whatever the stream job's watermark and window policy allow, which is a per-pipeline property rather than a platform one. "We are streaming" is not a freshness statement (Watermarks).
- In every arrangement, end-to-end freshness is set by the slowest hop, and calling a platform real-time because one hop is fast misleads every consumer who hears it (The Freshness SLO).
- The pattern itself evolves, and that migration is the expensive part. Moving from a central warehouse to domain ownership is not a data migration; it is a re-assignment of on-call, budget and definition authority, and the technical work is the smaller half.
- Consumers depend on the access path as much as the schema. Changing the arrangement usually changes connection strings, dataset names and permission models, so every downstream tool integration is in scope (Impact Analysis).
- Patterns compose incrementally, which is the good news: adding an event log beside a central warehouse is additive, and moving one domain to owning its own products is a pilot rather than a big bang (From a Simple App to a Scaled System).
- Recoverability is a property of raw retention and determinism, not of the pattern. If the raw arrival is intact and the transformations are pure functions of their inputs, every arrangement here is a re-run (Reprocessing vs Retrying).
- Event-driven and Kappa arrangements move the recovery horizon into log retention: you can rebuild anything the log still holds and nothing it does not, which turns retention from a cost decision into a recovery-window decision (Retention and Replay).
- Backing out of a pattern is the recovery nobody plans. Keep the previous arrangement's serving tables alive and populated through the migration so a rollback is a routing change rather than a rebuild (Rolling Back Data).
What can go wrong
- Adopting a pattern for a constraint you do not have, and inheriting its operational cost with none of its benefit.
- Arguing two patterns from different axes against each other until the decision is made by exhaustion.
- Building the boxes without the operating model, so ownership is nominal and every incident is routed to the platform team by default.
- A hybrid that is genuinely two architectures maintained in parallel because nobody was willing to decommission the first one (Data Platform Anti-Patterns).
- The mitigation failing: a written decision document with revisit triggers that nobody re-reads, so it becomes an argument from authority rather than from fit.
- "There is a best pattern." Each one is a response to a constraint. The useful question is which constraint binds you now, and the second-most useful is which one will bind you next year.
- "Mesh is the modern version of the central warehouse." They are answers on the ownership axis to different organisational sizes. A three-team company running mesh has invented overhead; a two-hundred-team company running a single central model has invented a queue (Data Mesh).
- "Lambda versus Kappa is a settled argument." It is settled only where your inputs are genuinely a replayable log. Where they are vendor file drops and API pulls, the batch path does not disappear because you renamed the architecture (Kappa Architecture).
- "We adopted the pattern, so the problem is solved." The pattern reallocates work. If the work was not resourced after the reallocation, the problem simply moved to a team that cannot see it (Who Owns Data Quality).
- "Storage choice and architecture pattern are the same decision." They are separable, and treating them as one is how organisations end up migrating storage in order to fix an ownership problem (The Data Warehouse).
- Each pattern implies a different enforcement point for classification, retention and access. Centralised modelling gives one place to enforce them; distributed ownership requires the policy to be computational — expressed as code the platform applies — or it will be applied inconsistently by design (Data Classification).
- Deletion obligations cross every pattern boundary. A subject-deletion request must reach the log, the raw layer, every domain product derived from it and every extract, and no arrangement here makes that automatic (Deletion Requests).
Operating it
- Time from "a team wants a new dataset" to "a governed dataset exists". This single number distinguishes the ownership patterns better than any diagram, and it is measurable from your ticket system today.
- Number of datasets with a named owner who has responded to an alert in the last quarter, versus datasets with an owner field filled in. The gap is the honest ownership picture (Data Ownership).
- Count of distinct definitions in production for your three headline metrics. Zero divergence suggests central modelling or good federated governance; several suggests neither (The Metrics Layer).
- Per-dataset freshness against its stated SLO, published where consumers see it — the only freshness signal that is not an average hiding a broken table (The Freshness SLO).
- At 10x data volume none of these patterns changes; the physical layout and incremental processing decisions do, and they are orthogonal (Physical Data Layout).
- At 10x source count, the central modelling team stops having domain knowledge for what it is modelling, and the ownership axis starts mattering more than anything technical.
- At 10x consumer count, discoverability and access control become the binding constraint, and a catalog stops being optional under every pattern (The Data Catalog).
- Below roughly one platform team and a handful of source systems, all of these patterns reduce to "a warehouse and some scheduled SQL", and choosing among them is a way of avoiding work.
- The dominant cost of an architecture pattern is coordination, and it does not appear on any infrastructure bill. Meetings to agree a definition, tickets waiting in a queue, and the same model built twice by two teams are all real cost with no line item (Cost Attribution).
- Duplicated computation is the second driver, and it differs sharply by pattern: Lambda computes the same logic twice by design, an event-driven platform lets every consumer recompute the same derivation independently, and a central warehouse computes it once (Compute Waste).
- Retained bytes shift with the pattern. Log-centric arrangements pay storage for the recovery window they promise; batch-centric ones pay it in raw and intermediate layers (What Actually Drives Data Platform Cost).
- Migration cost between patterns is dominated by consumer coordination rather than by moving data, and it scales with the number of tools that hardcoded a connection (Impact Analysis).
- Centralising meaning costs throughput; distributing it costs coherence. Every pattern here is a position on that line, and there is no position that is free of both costs.
- Every additional dataflow path — a speed layer, a second engine, a replay path — buys a capability and costs an implementation that must stay consistent with the others forever.
- Choosing a pattern with a written operating model costs weeks of unglamorous agreement up front and saves the years in which nobody knows who owns a broken table.
Architecture axes
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
| Batch ↔ Streaming | |
|---|---|
| Batch buys | Re-runnable, testable, and one implementation. Most reporting decisions are made after the period anyway. |
| Streaming buys | A number before the period ends, for a decision that genuinely cannot wait. |
| Decided by | Whether anyone acts on the number inside the period. If not, streaming is cost with no buyer. |
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe two-axis framing — dataflow arrangement versus ownership arrangement — is stable across stacks and eras, because both axes describe organisational and computational commitments rather than products. What varies is which point on each axis a given toolchain makes cheap.
- ORG-SPECIFICEverything on the ownership axis is decided by team count, team capability and who controls budget and on-call. The same technical platform supports central and distributed modelling; only the organisation decides which one is actually happening, and a reorganisation can change the answer without a line of code.
- SCALE-SPECIFICBelow a few source systems and one platform team these patterns collapse into a single sensible answer and the comparison is wasted effort. They separate once no single team can hold the domain knowledge of every source, which is a team-count threshold rather than a byte-count one.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the guarantees every one of these arrangements inherits — delivery semantics, ordering across machines, and what a replay actually reproduces. When that domain lands, the
guaranteesfield of each pattern here should link into it. - — DevOps / Production Engineering owns the delivery half of an architecture decision: how a migration between patterns is rolled out without a flag day, and how ownership is expressed as on-call rather than as a diagram.