What Data Engineering Actually Is
Not the tools. The discipline of moving data between systems so that what arrives is complete, correct, explainable and affordable.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A dashboard says revenue was €1,245,892 yesterday. What had to be true for that number to be trustworthy?
Everyone downstream of an operational database: analysts writing SQL, finance closing a month, a product team measuring an experiment, a model training on last year, an agent retrieving a document. None of them can see your pipeline. All of them see its output and assume it is the truth.
The unit of this domain is the record in transit — one order row, one click event, one CDC change, one file. Everything that follows is a question about what happens to that record between the system that produced it and the system that reports on it.
Point the BI tool at a read replica of the production database and write SQL against the application's tables. There is no pipeline, no duplication, no staleness, and no second system to operate. For a small product this genuinely works, and any advice that skips over that is selling something.
The first analytical query that scans two years of orders takes the replica down, or takes long enough that the replica lags, and the application starts serving stale reads from it (Read Replicas From the Application).
- The first analytical query that scans two years of orders takes the replica down, or takes long enough that the replica lags, and the application starts serving stale reads from it (Read Replicas From the Application).
- The application team renames
amount_centstoamount_minorin a migration. Nobody told the analysts, because nobody knew analysts were reading that column. Every revenue dashboard silently reports zero, and the schema change was correct. - Someone asks "what did this customer's tier look like when they placed that order?" The operational table holds only the current tier. The information required to answer that question was overwritten months ago and cannot be recovered (Slowly Changing Dimensions).
- A hard delete for a GDPR request removes the row. The finance report for the quarter it belonged to now produces a different number than it did last week, and neither number is reproducible.
- A second source appears — the payment provider, a SaaS CRM, the event stream from the mobile app — and none of it lives in that database. The query that joins them has nowhere to run.
What is actually happening
- Operational systems are optimised for a workload that is the opposite of the analytical one: many small transactions touching a few rows of *current* state, with low latency and strict correctness (OLTP vs OLAP). Analytical queries scan enormous ranges of *historical* state and aggregate. The same storage layout cannot serve both well.
- So the data moves. Every stage of that movement is a copy, and every copy introduces the possibility of divergence: rows that did not arrive, rows that arrived twice, rows that arrived late, rows whose type or meaning changed on the way.
- Data engineering is the discipline of making those copies trustworthy on purpose rather than by luck. It is a guarantees problem before it is a tooling problem: what does each hop promise, and what does it explicitly not promise (The Data Loop)?
- The two properties that distinguish this from ordinary backend work are history and reproducibility. An operational system is allowed to forget; an analytical one is asked what was true in March. A request handler answers once; a transformation is expected to produce the same answer when re-run next year.
- The failure mode is also different in kind. A backend failure is loud — an error rate, a page, an angry user. A data failure is quiet: the job succeeded, the table has rows, the dashboard renders, and the number is wrong. You find out from a person, not a monitor (The Pipeline Succeeded. The Data Is Wrong.).
The journey a single order takes
A customer taps "Buy". The backend opens a transaction, inserts a row into orders, commits, and returns a confirmation. From the application's point of view the story ends there. From this domain's point of view it has just begun, because that row now needs to become a number in a report, and it has a long way to travel.
Each hop below is a separate system with its own failure behaviour, its own retry semantics and its own idea of what it promises. The chain is only as strong as the weakest promise in it, which is why the guarantees column matters more than the does column.
Notice what is *not* in the chain: any step that verifies the number at the end matches the row at the beginning. That step does not exist unless you build it, and its absence is why most data incidents are discovered by the person reading the dashboard.
- 1Application
Writes the order inside a database transaction.
guarantees Atomicity and durability of that row, and nothing at all about anything downstream.
fails by Committing business logic that is itself wrong — the strongest guarantee in the chain is about storage, not meaning.
- 2Change capture
Reads the committed change from the database log and emits it as an event.
guarantees Every committed change is emitted at least once, in commit order per source.
fails by Falling behind and being cut off by log retention, at which point changes are gone rather than late.
- 3Event log
Stores the change durably and lets many consumers read it independently.
guarantees Durable, replayable, ordered within a partition. Not ordered across partitions.
fails by Retention expiring before a consumer catches up; a key whose partition changed losing order against its own history.
- 4Raw landing
Writes arriving events to object storage, untouched.
guarantees What arrived is preserved exactly, so anything downstream can be rebuilt.
fails by Being "cleaned" on the way in, which destroys the only copy that could have proved what really happened.
- 5Transformation
Cleans, joins, aggregates and models into fact and dimension tables.
guarantees Only what its tests assert. By default: nothing.
fails by A join that fans out rows, a filter that drops a category, a cast that silently nulls — all of which succeed.
- 6Validation
Runs data tests before publishing.
guarantees That the assertions you wrote hold. Never that the data is correct.
fails by Passing every test while the metric is meaningless, because no test encoded the meaning.
- 7Serving table
Holds the modelled result for query.
guarantees Query performance and, if published atomically, that readers never see a half-written state.
fails by Being overwritten by a backfill while consumers read it.
- 8Dashboard
Aggregates the serving table into a number a human reads.
guarantees Nothing. It renders whatever it is given, with total confidence.
fails by Applying its own filter or join that changes the metric, invisibly, in the BI tool.
Read the guarantees column top to bottom. The strongest promise is at the source and it weakens at every hop — which is the entire reason this domain exists.
What can go wrong between the source and the dashboard
Ask an engineer what breaks in a data pipeline and you will usually hear "the job fails". Job failures are the easy case: they are loud, they are attributable, and the orchestrator tells you. The expensive failures are the ones where everything succeeds.
The list below is the domain's failure taxonomy, and most of the rest of these lessons are a long answer to one entry in it. What matters at this stage is recognising that each has a different detection method — there is no single check that finds them all, and a platform that only monitors task status finds almost none of them.
- Missing rows — an extract window that excluded them, a CDC gap, a filter that was too broad. Detected by reconciliation against the source, not by anything internal (Missing Rows).
- Duplicate events — at-least-once delivery, a retried job, a re-run without idempotency. Detected by a uniqueness test on the business key (Duplicate Rows).
- Schema changes — a field added, removed, renamed or retyped upstream. Detected by contract enforcement at the boundary (Schema Evolution).
- Late data — an event that happened yesterday and arrived today, after yesterday's partition was already computed (Late-Arriving Data).
- Out-of-order events — arrival order that does not match event order, so a "latest state" computed by arrival is wrong (Event Time).
- Bad types — a numeric field arriving as a string, a timestamp without a zone, a cast that produces null rather than an error (Nullability & Defaults).
- Broken transformation — a join that multiplies rows, a
GROUP BYat the wrong grain, a window function with the wrong frame (Grain: What Does One Row Represent?). - Stale data — the pipeline has not run, or has run and produced nothing, and the dashboard shows a confident number from last Tuesday (Freshness Checks).
- Partial loads — half the partitions published, so a consumer reads a period that is genuinely incomplete and looks merely low (Atomic Publish).
- Backfills — a historical recompute that overwrote current data, or double-counted, or ran against a source that has since changed (What Backfills Break).
- Partition skew — one key holding most of the data, so one task decides the runtime of the whole job (Data Skew).
- Pipeline failure — the loud case. Worth listing precisely because it is the one everybody already monitors.
- Incorrect business logic — the transformation does exactly what it was told and what it was told was wrong. No technical check will ever find this; only a consumer who knows the domain will (Two Dashboards, Two Numbers).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Row count per period versus source | Completeness — everything that happened arrived. | Missing rows, dropped partitions, an extract window that closed early, a filter that was too broad. | Duplicates that coincidentally offset losses; any period that is not yet closed; wrong values in rows that are all present. |
| Uniqueness on the business key | Each real-world event appears exactly once. | At-least-once redelivery, a re-run without idempotency, a fan-out join. | Duplicates that differ in the key — the same order re-emitted with a new event id looks like two orders. |
| Freshness: newest complete record versus now | The data is recent enough for the decision it drives. | A pipeline that stopped, a source that stopped, an upstream that is silently empty. | Fresh data that is wrong; and it fires falsely on any period where the source genuinely produced nothing. |
| Daily volume against its own history | Today looks like a normal day. | A partial load, a broken filter, a source outage, an upstream schema change that nulled a column used in a WHERE clause. | Slow drift; any error that preserves volume, which includes almost every value-level bug. |
Every row's misses column is the reason no single check is sufficient. Quality is a portfolio, and the portfolio is chosen from what each consumer would actually notice.
Why "the pipeline succeeded" proves nothing
An orchestrator reports on task execution: did the process exit zero, within its timeout, without raising. That is a statement about code, and it is genuinely useful — but it is orthogonal to whether the data is right.
Consider a transformation that reads yesterday's partition, joins it to a customer dimension, and writes a fact table. If the upstream extract wrote zero rows because the source API changed its pagination, the join succeeds, the write succeeds, the task succeeds, and the fact table gains zero rows. Green everywhere. The dashboard shows yesterday as a very quiet day.
Now consider the opposite: the customer dimension gained duplicate rows because a re-run was not idempotent. The join now emits several rows per order. The task succeeds — faster than usual, even, because there is no error to raise — and revenue is reported as several times its real value. Green everywhere, and this time the number is high, which people question far less often than a number that is low.
This is the observation the whole domain is built around, and it is why every lesson here carries a quality field. The question is never "did it run". It is "what would have to be true for this output to be right, and which of those things am I actually checking".
Alert when a DAG task fails or exceeds its runtime. Page the on-call engineer. Re-run the task. Close the incident when the task is green.
Alert when a serving dataset's freshness exceeds its stated SLO, when its row count deviates from its own history, when a uniqueness or completeness test fails, or when reconciliation against the source diverges. The task being green is one input among several.
Task status can only report failures that raise. The failure classes that matter most in this domain — missing rows, duplicates, late data, semantic drift — all produce successful runs by construction, so a monitor watching task status is structurally unable to see them.
How to build it
Most important first.
- Start from the consumer, not the source. Which questions must be answerable, at what freshness, with what history? Those three answers determine almost every architectural choice that follows, and reasoning in the other direction produces a platform full of datasets nobody uses.
- Separate the analytical workload from the operational one before it is urgent — the moment analytics can affect production availability, the two systems are coupled in the worst possible way (Workload Isolation).
- Keep the raw arrival immutable. Transformations will be wrong; the ability to reprocess from an untouched original is what makes that survivable rather than terminal (Keeping Raw History: The Recovery Position and the Liability).
- Make every step re-runnable without changing the result. Idempotency is not an advanced topic here — it is the precondition for ever fixing anything (Idempotent Data Pipelines).
- Write down what each dataset promises: its grain, its freshness, its owner, its known gaps. An undocumented dataset is not an asset, it is a liability with rows in it (Data Contracts).
- Instrument the data, not just the pipeline. Row counts, freshness, null rates and distribution shape are the signals that catch the failures a task-level "success" cannot (Data Observability).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The operational database guarantees, at commit, that the row is durable and consistent with its constraints. That is the strongest guarantee anywhere in the chain, and it applies only inside that system.
- Everything downstream inherits at most what its weakest hop promises. A warehouse table fed by an at-least-once stream is an at-least-once table, however transactional the warehouse is (At-Least-Once Delivery).
- No hop in a typical pipeline guarantees completeness by itself. Completeness is something you *measure* by reconciliation against the source, not something you receive (Reconciliation).
- Nothing guarantees that the meaning of a field is unchanged. Types are checked; semantics are not (Semantic Changes).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The first check any platform should have is a reconciliation: count rows and sum a monetary column in the source for a closed period, compare with the same aggregate in the serving table, and alert on divergence. It catches missing rows, duplicated rows and bad joins in one query.
- It misses everything about a period that is not yet closed, anything where source and destination are wrong in the same way (a bug in shared logic), and any error in a column it does not sum.
- It also cannot tell you the number is *meaningful* — a perfectly reconciled
revenuecolumn that changed from gross to net reconciles beautifully and reports the wrong thing.
- Reading a replica directly is the freshest possible analytical answer and the least isolated — you trade availability for latency in the most direct way available.
- Every stage added between source and consumer adds a place where data can sit. The end-to-end freshness a consumer experiences is the sum of every hop's delay plus the schedule interval of the slowest one, not the speed of the fastest.
- The right question is never "how fresh can we make it" but "what decision does this number drive, and how stale can it be before that decision changes". Most dashboards that demand minutes are read once a day.
- The source schema will change and you will usually not be told. That is not a process failure to be fixed once; it is the steady state to be designed for (Schema Evolution).
- The safest posture is to land raw data in a form that tolerates unknown fields, and to fail loudly at the point where a transformation makes an assumption — early enough to catch it, late enough not to reject data you could have kept.
- Semantics change without the schema changing at all, and no type system will catch it. That is what data contracts and column-level documentation are for (Data Contracts).
- The entire recoverability of a data platform reduces to one question: can you rebuild the serving tables from something you still have? If raw is retained and transformations are deterministic, almost any mistake is a re-run.
- If raw was transformed in place, or the source has since been mutated, or the transformation depends on
now(), then the mistake is permanent and the only remaining option is an apology. - Design the re-run before you need it: bounded by an explicit range, writing to a location consumers are not reading, validated before publish (Planning a Backfill).
What can go wrong
- The pipeline runs successfully and produces wrong data. This is the characteristic failure of the domain and the reason task-level monitoring is insufficient.
- A schema change upstream that types cleanly and means something different.
- An analytical query taking capacity from the operational system it reads.
- History overwritten before anyone realised it was needed — the one failure with no recovery path.
- Two teams computing the same metric two ways, both defensibly, and executives receiving both (The Metrics Layer).
- A monitoring system that watches the orchestrator and not the data, so every incident is reported by a human.
- "Data engineering is ETL." ETL is one activity inside it. Modeling, contracts, quality, lineage, governance and cost are not transformation steps and are usually where the value is.
- "We need a warehouse." Perhaps. First say which question is not currently answerable and why. A surprising number of warehouse projects are launched to solve a problem an index would have solved.
- "The data team owns data quality." The team that produces a field owns whether it is correct; a data team can only measure and report. Placing the whole obligation downstream guarantees it fails (Who Owns Data Quality).
- "Our dashboards match, so the data is right." Two dashboards agreeing means they share an upstream, which is exactly what you would expect if that upstream is wrong.
- The moment data is copied out of the operational system, the copy inherits every obligation of the original — access control, retention, deletion — and none of the mechanisms that enforced them (Data Governance).
- A deletion request against a source row does not delete it from the lake, the warehouse, the extract in someone's notebook, or the model trained on it. Designing for that on day one is far cheaper than discovering it during an audit (Deletion Requests).
Operating it
- Freshness per serving dataset: the gap between now and the newest complete record it contains. One number, per table, that a consumer can see (Freshness Monitoring).
- Row counts per run compared with the same weekday historically — a volume anomaly is the cheapest broad detector there is (Volume Anomalies).
- A lineage graph from the dashboard metric back to the source table, because during an incident the first question is always "what feeds this" and the second is "what else does it feed" (Data Lineage).
- At 10x volume the read-replica approach is normally gone already, and the questions become layout ones: how is data partitioned, how much does a query scan, how long does the nightly rebuild take.
- At 100x, full rebuilds stop being possible within the schedule and everything must become incremental — which introduces state, watermarks and late data as first-class problems (Incremental Processing).
- Consumer count scales the *governance* problem rather than the technical one. Ten datasets and three consumers need no catalog; five hundred datasets and eighty consumers cannot function without one (The Data Catalog).
- The dominant costs in a data platform are bytes scanned by queries, bytes moved across the network by shuffles, bytes retained over time, and work repeated because it was not incremental (What Actually Drives Data Platform Cost).
- Every one of those is a design decision made months earlier — the layout, the model, the schedule — and each is much cheaper to change before a hundred dashboards depend on it.
- The cheapest data platform is one that does not exist. Reaching for a warehouse when a read replica and a scheduled report would do is the single most common over-build in the field.
- A data platform is a second copy of your company's data with its own failure modes, its own on-call, its own cost line and its own security surface. It buys isolation, history and query power; it costs an ongoing operational commitment that never ends.
- Every guarantee added — exactly-once effects, full reconciliation, column-level lineage — is bought with latency, storage or engineering time. Buying all of them everywhere is how platforms become too expensive to justify.
- Starting simple means migrating later. Starting sophisticated means paying for capability you may never need. There is no third option, and the honest answer is to start simple and keep raw data so the migration is possible.
Follow one data point
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
A person, on a phone, with an intermittent connection, presses a button once. They believe they have bought one thing.
One human intent. This is the only place in the entire chain where the grain is unambiguous, and everything downstream is an attempt to preserve it.
guarantees Nothing yet. The intent exists only in the user's head and in a pending HTTP request.
The request may be sent more than once — by the client library on timeout, by the user pressing again, or by a proxy replaying it. The number of clicks and the number of requests are different quantities from here onward.
- ·The connection drops after the request is sent and before the response returns, so the client retries and the intent arrives twice.
- ·The user presses again because they saw an error, producing a second intent that is genuinely a second intent — indistinguishable, downstream, from a duplicate.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe journey from operational write to analytical read, and the failure classes along it, hold regardless of stack. What changes between organisations is which hops exist, not what each hop can lose.
- SCALE-SPECIFICBelow roughly one product, one database and a handful of analysts, a read replica with scheduled reports is a legitimate answer and a warehouse is overhead. The advice inverts once analytical queries can affect production availability or a second source appears.
- ORG-SPECIFICOwnership, contracts and catalogs solve coordination problems that only exist above a certain number of teams. In a five-person company they are ceremony; at fifty they are the difference between a platform and a swamp.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the guarantees these hops inherit — delivery semantics, ordering, consensus and the impossibility results underneath them. That domain is being built separately; when it lands, every
guaranteesfield here should link into it. - — DevOps / Production Engineering owns how the transformation code itself is tested, versioned, deployed and rolled back. A data model is software, and it deserves the same delivery discipline.