Data Engineering vs Its Neighbours
Databases own storage, distributed systems own guarantees, backends own the transactional service, analytics and ML are consumers. We own movement, transformation, modelling, validation and serving.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A team argues about who owns a broken number — the backend that produced the event, the database it was written to, or the model that aggregated it. Where does each boundary actually fall?
An engineer deciding what to learn next, a manager deciding what a data team is accountable for, and — most concretely — the people in an incident review deciding whose backlog the fix goes into. Bad boundaries produce a data team that is either a SQL service desk or a second infrastructure team, and neither builds a platform.
One unit here is one responsibility: a decision that some team must make and be answerable for. Responsibilities are the right grain because they are what actually gets assigned — you cannot assign "databases" to anyone, but you can assign "who decides the retention on the raw layer" and "who is paged when fct_orders is stale".
Define the data team by its tools: they are the people who use Airflow, Spark and dbt. It is an honest description of what the team touches on a Tuesday, and it produces an unambiguous org chart.
The moment the warehouse ships a scheduler, or the backend team writes a dbt model, the tool boundary stops matching the responsibility boundary and nobody knows who is on call for fct_orders (Data Ownership).
- The moment the warehouse ships a scheduler, or the backend team writes a dbt model, the tool boundary stops matching the responsibility boundary and nobody knows who is on call for
fct_orders(Data Ownership). - A tool-defined team inherits every request that mentions its tools, including "make this dashboard pretty" and "give me a CSV", and loses the ones that matter, like "what should this column mean" (Who Owns Data Quality).
- It leaves the most consequential decisions unowned. Nobody owns whether
customer_countryis authoritative in the CRM or the warehouse, because that is not a tool question (Source of Truth). - It invites the team to re-solve its neighbours' problems badly — hand-rolled consensus, a bespoke cache, a second metrics store — because the tool boundary gives no reason to stop (Data Platform Anti-Patterns).
- It makes learning unbounded. "Learn data engineering" becomes "learn every tool that has ever touched data", and the primitives that actually transfer never get studied (The Data Loop).
What is actually happening
- Draw the boundary by the question each domain answers, not by the software each uses. Database Engineering answers how data is stored, indexed, queried, transacted and replicated *inside one system*. Data Engineering answers how it moves *between* systems and what survives the trip.
- Distributed Systems answers what is guaranteed across machines — delivery, ordering, consistency, and the impossibility results underneath. We are that domain's largest customer: every arrow in a pipeline inherits a guarantee we did not invent and must not overstate.
- Backend Engineering answers how the application-facing transactional service is built and operated. We are strictly downstream of it: we read its tables, its log and its events, and we do not get to change how it writes (What a CDC Event Contains).
- Analytics/BI and ML/AI are consumers, not neighbours. The distinction matters because a consumer relationship has a contract — a grain, a freshness, a schema, an owner — and a peer relationship does not. Blurring it is how data teams end up writing dashboards and training models instead of serving the people who do (Who Actually Consumes This Data).
- The overlaps are real and they are where the arguments happen. Physical layout is a database concept applied to files. Schema evolution is an API-versioning problem applied to tables. Freshness monitoring is observability applied to rows rather than requests. In each case the *technique* is borrowed and the *object* is ours.
Nine boundaries, stated precisely
Every row below has the same shape: the neighbour's question, ours on the same subject, and the overlap that actually causes arguments. The third column is the useful one — nobody argues about the clear cases.
Read the middle column as a constraint on scope. If a piece of work does not fit any entry in it, that work belongs to a neighbour and the correct move is to link, ask or delegate rather than to build a lesser version of what they already have.
Two rows are marked as consumers rather than neighbours. That is the single most consequential distinction on this page: a neighbour is someone you negotiate with, and a consumer is someone you serve under a contract you publish (Data Products).
| Domain | Its question | Ours on the same subject | The overlap people argue about |
|---|---|---|---|
| Database Engineering | How is data stored, indexed, queried, transacted and replicated inside one system? | How does data move *between* systems, and what is transformed, modelled, validated and served on the way. | Physical layout. Partitioning and file size are database ideas applied to files in object storage, and both domains have a legitimate claim (Physical Data Layout). |
| Distributed Systems | What is guaranteed across machines — delivery, ordering, consistency, consensus — and what is impossible? | Which of those guarantees a pipeline actually depends on, and what breaks when the assumption does not hold. | Delivery semantics. Saying "the log gives us ordering" without saying per what is the most common piece of guarantee inflation in this field (CDC Ordering and Transaction Boundaries). |
| Backend Engineering | How is the application-facing transactional service built and operated? | We are its downstream consumer: we read its writes, its log and its events, and we do not get to change how it writes. | Who owns the event schema. The backend emits it; we depend on it; nobody owns it until a contract says so (Data Contracts). |
| Cloud & Infrastructure | Where does it run, on which primitives, at what operational cost? | Which storage and compute primitives a data platform is assembled from, and what each one's shape implies for layout. | Storage and compute separation, which is simultaneously an infrastructure property and the defining architectural choice of modern analytics (Separating Storage from Compute). |
| DevOps / Production Engineering | How is software delivered, versioned, deployed and rolled back? | How *data models, schemas and pipelines* are versioned, deployed and rolled back — the same discipline applied to a different artefact. | Rollback. Rolling back code is well-understood; rolling back a published dataset means republishing history and telling people the number changed (Rolling Back Data). |
| Observability & Performance | Why is it slow, where is the bottleneck, is the service healthy? | Is the data correct, complete and fresh — a different question, with a different toolkit, that borrows the same vocabulary. | The word "latency". Request latency and dataset freshness are both measured in time units and are otherwise unrelated (Data Observability). |
| System Design | How do components fit together to meet a requirement? | How the *data plane* specifically fits together, where the copies live and what each hop promises. | Architecture diagrams that show a warehouse as one box, which hides every decision this domain exists to make (Data Architecture Patterns). |
| Analytics / BI | — consumer — | We serve them: modelled tables at a stated grain, with stated freshness, and a metric definition they can rely on. | Who writes the dashboard and who owns the metric definition. These are different jobs and are constantly merged (The Metrics Layer). |
| ML / AI | — consumer — | We serve them: training sets, feature pipelines, embedding corpora and evaluation data, each with lineage and reproducibility. | Feature computation, which is modelling work with a training-serving consistency requirement that pure analytics does not have (Feature Pipelines). |
Consumers are not neighbours
A neighbouring domain is a peer: you negotiate, you divide work, you occasionally disagree about who builds a thing. A consumer is on the other side of an interface you publish, and the whole relationship is defined by what that interface promises.
The distinction is not politeness. It changes what you build. For a peer you build integrations; for a consumer you build a contract — a stated grain, a stated freshness, a schema with a compatibility policy, an owner and a documented list of known gaps. Everything in this domain that looks like bureaucracy is the machinery of that contract (Dataset Documentation).
It also changes how you decide. Designing from the source outward produces datasets that reflect whatever shape the upstream happened to have. Designing from the consumer inward starts from the questions that must be answerable, at what freshness, with how much history — and those three answers determine most of the architecture (Who Actually Consumes This Data).
The diagram below is the whole positioning in one picture. Producers on the left, consumers on the right, and every guarantee in the middle is something this domain either provides or fails to.
Who owns this? A test you can run in a meeting
Boundary arguments are almost always about a specific artefact — an event schema, a stale table, a slow query, a wrong number — and they resolve quickly if you ask what kind of decision is being made rather than which tool is involved.
The four options below are not a hierarchy and there is no default. What makes the test work is that each option comes with a cost, so choosing one is visibly choosing to pay something rather than to win an argument.
One warning about the last option. "Nobody owns it yet" is a legitimate answer in a small organisation and a dangerous one in a large one, because unowned datasets do not stay unowned — they get depended on, and then they are owned by whoever is nearest when they break (Data Ownership).
What kind of decision is actually being made about it?
when The decision is about whether a field is correct, what it means, or when it is emitted. Nothing downstream can fix a wrong source.
cost The producer takes on a downstream obligation they did not previously have, and needs a contract and a test to make it concrete rather than a request to be careful (Data Contracts).
when The decision is about movement, transformation, modelling, validation or serving — whether the record arrives, once, in order, typed and at the declared grain.
cost On-call for datasets, an obligation to publish freshness and grain, and the discipline to refuse work that is really analytics or really infrastructure (The Freshness SLO).
when The decision is about what question is being asked, which definition is correct for their purpose, or how a number is presented.
cost They must state their freshness need, their tolerance for revision and their definition explicitly — which is harder than asking for "the data" and is the only way to get served well (Who Actually Consumes This Data).
when The decision is about storage internals, consensus, deployment mechanics or the attacker's view. Borrow the technique; do not re-derive it.
cost A dependency and a hand-off, plus the obligation to learn enough of that domain to state what your pipeline relies on (Data Engineering and Distributed Systems).
How to build it
Most important first.
- State the boundary as an ownership sentence for each neighbour: "the backend owns whether the event is emitted; we own whether it arrives, once, in order, and typed". Ambiguity in that sentence is the source of every escalation later (Data Contracts).
- Deep-link rather than duplicate. A data lesson that explains B-tree internals, Raft or TLS handshakes has wandered; link to the domain that owns the depth and keep the focus on which of its guarantees your pipeline depends on (Data Engineering and Database Engineering).
- Treat consumers as contract counterparties. Publish grain, freshness, owner and known gaps for every serving dataset, and let the consumer's needs drive the design rather than the source's shape (Data Products).
- Borrow the neighbours' hard-won discipline explicitly: version control, review, testing and rollback from delivery engineering; SLOs and error budgets from observability; compatibility rules from API design. None of these need reinventing (Data Engineering and DevOps).
- Know exactly which distributed-systems guarantee each hop relies on, and write the one you do *not* have next to it. Most pipeline design errors are guarantee inflation, not missing features (Data Engineering and Distributed Systems).
- Push responsibilities upstream where the upstream team is better placed to hold them. The producer of a field owns whether it is correct; a data team can measure and report, and no amount of downstream heroics fixes a wrong source (Who Owns Data Quality).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- This lesson guarantees nothing technical — it is a boundary, and boundaries are organisational agreements. What it buys is that during an incident, one team is answerable per hop rather than three teams being partly answerable for all of them.
- The boundary does not survive contact with a reorg unless it is written down as ownership of *datasets and hops*, not of tools. Tool-based ownership is re-litigated every time a vendor ships a feature (Data Ownership).
- Nothing here promises the neighbours agree with your version of the boundary. Getting the producing team to accept "you own whether the field is correct" is a negotiation, and it usually needs one incident behind it.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that tests the boundary is an incident review: for the last five data incidents, was the owning team obvious within the first ten minutes? If not, the boundary is described but not real.
- A second check is dependency direction. If the backend team has to ask permission from the data team to change a column, the boundary is inverted — contracts should make the producer's obligations explicit without making them a downstream approval queue (Contract Enforcement).
- Both checks miss the slowest failure: a boundary everyone agrees on and nobody staffs. Ownership without capacity looks identical to ownership on paper and produces the same outcome as no ownership at all.
- The boundary decides who is on the hook for freshness. If the backend owns emission and we own arrival, a stale dataset caused by a paused producer is a joint incident with an unambiguous first responder (The Freshness SLO).
- Analytics consumers experience freshness as the end-to-end number and rarely know which hop dominates it. Publishing per-dataset freshness rather than a platform-wide figure is what makes the boundary legible to them (Freshness Monitoring).
- Cross-domain: latency in the request path is Observability and Performance's object; latency of a *dataset* is ours. They use the same word and measure different things, and conflating them wastes a lot of meeting time.
- The boundary moves as products absorb each other. Warehouses gained schedulers, streaming engines gained SQL, table formats gave files transactions — each shift moved a responsibility across a line without moving the question that defines it (Open Table Formats).
- That is precisely why the boundary is drawn by question rather than by product. "Who owns transformation scheduling" has a stable answer; "who owns Airflow" does not survive the next release note.
- Consumer relationships evolve too. ML teams that once received CSV extracts now depend on feature pipelines with freshness SLOs, which converts an informal hand-off into an operated dataset (Feature Pipelines).
- When a boundary turns out to be wrong, fix it in the ownership record and the on-call rotation first, not in the architecture. Most boundary problems are answered by moving a name, not by moving a system.
- When a data team has drifted into being an analytics team, recovery is slow and starts by handing back the dashboards, with a serving model good enough that the hand-back is not a downgrade (Data Marts).
- When it has drifted into being an infrastructure team, recovery starts by adopting a managed primitive for whatever it built by hand, and re-spending the time on modelling and contracts (Data Platform Engineering).
What can go wrong
- The data team becomes the analytics team, because dashboard requests are urgent and modelling is not, and within a year nobody owns the platform.
- The data team becomes a second infrastructure team, operating brokers and clusters that a managed service would have provided, and the modelling and quality work never starts (Data Platform Anti-Patterns).
- The backend team treats every downstream break as the data team's problem, so contracts are never enforced and schema changes stay a surprise (Breaking Schema Changes).
- The boundary is defended too hard: "distributed systems owns that" becomes a reason not to understand ordering, and a pipeline is built on a guarantee nobody checked (CDC Ordering and Transaction Boundaries).
- ML teams build a shadow pipeline because the platform did not serve them, and the company acquires two definitions of every metric (Two Dashboards, Two Numbers).
- "Data engineers do not need to understand databases." The opposite: this domain reads a database's log, depends on its commit ordering, and inherits its transactional semantics. It is the neighbour you can least afford to treat as a black box (Data Engineering and Database Engineering).
- "A data warehouse is just a bigger database." It is a different workload with a different storage layout, a different execution model and different economics. The word "database" is shared; almost none of the engineering is (The Data Warehouse).
- "Analytics engineers are just data engineers who write SQL." They are usually best understood as owning the modelling and metrics boundary — the part closest to the consumer — which is a real specialisation and not a subset (Analytical Data Modeling).
- "Data mesh means every team runs its own platform." It is an organisational model for ownership, with a real operational cost, and it usually still requires a central platform underneath it (Data Mesh).
- "We are downstream, so quality is not ours." We own detection and we own not making it worse. We do not own the correctness of a field we did not produce, and pretending otherwise guarantees the producer never fixes it.
- Security Engineering owns the attacker's view — how a system is broken into, how credentials are stolen, how a boundary is crossed. We own classification, minimisation, retention and access as they apply to *datasets and pipelines*, which is a distinct object with distinct mechanisms (Data Governance).
- The obligation that follows a copy is ours specifically because we made the copy. A deletion request satisfied in the source is not satisfied in the lake, the warehouse, the extract or the trained model, and no neighbouring domain is watching for that (Deletion Requests).
Operating it
- The on-call rotation. Which team is paged for a stale serving dataset, and whether that has ever been ambiguous, is the most direct measurement of whether the boundary exists.
- The dataset ownership field in the catalog, and how many datasets have one. A catalog where most entries are unowned describes a platform with no boundary at all (The Data Catalog).
- The ratio of ad-hoc requests to platform work in a data team's tracker. A team above a certain ratio has become a service desk regardless of what its charter says.
- At small scale the boundary is mostly notional: one team does the backend, the pipeline and the dashboard, and that is efficient rather than sloppy.
- It starts mattering at the point where the producer and the consumer of a dataset are different people who do not attend the same standup. That threshold is organisational and arrives well before any technical one.
- At large scale the boundary becomes the platform's product: central teams own primitives and contracts, domain teams own datasets, and the interesting failures are all coordination failures (Data Mesh).
- Boundary confusion costs coordination, not compute — duplicated pipelines, duplicated metrics, and the meetings that discover the duplication months later (Cost Attribution).
- Rebuilding a neighbour's primitive is the expensive version: a self-operated broker or a hand-rolled catalog costs engineering time permanently and is rarely better than the managed alternative at the scale it was built for.
- A clear boundary lets cost be attributed to the team whose decisions drive it, which is the precondition for anyone actually reducing it.
- A sharp boundary makes ownership clear and makes hand-offs slower. Every contract is a coordination cost paid on every change, and a small organisation can genuinely go faster without one.
- Refusing analytics work protects the platform and disappoints the people asking. That disappointment is real, and it is only survivable if the serving layer is good enough that they can answer their own questions (The Self-Service Data Platform).
- Deep-linking rather than duplicating means engineers must actually learn the neighbouring domain when they hit its edge. That is a higher bar than a self-contained playbook and it is the only version that produces judgement.
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe question each domain answers is stable across companies and decades — storage versus movement, guarantees versus dependence on them, production versus consumption. Which team holds which question is entirely local and changes with every reorg.
- ORG-SPECIFICBelow the point where producer and consumer are different people, these boundaries are overhead and one team doing everything is correct. Above it, the absence of a written boundary produces duplicated metrics and unowned datasets within about a year.
- SIMPLIFIEDThe table presents each domain as owning one question; real domains overlap heavily and the overlaps are where the interesting engineering is. Physical layout, schema compatibility and freshness monitoring each belong to two domains at once and are argued about accordingly.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems is the neighbour this domain depends on most and duplicates least. When it lands, the "guarantees" row of the table above becomes a set of links rather than a paragraph, and every
guaranteesfield in this domain should point into it. - — DevOps / Production Engineering owns delivery, versioning and rollback as a discipline. This domain applies that discipline to a different artefact — a model, a schema, a published dataset — where rollback means republishing history and telling people the number moved.