Data Engineering Anti-Patterns
Sixteen decisions that were reasonable when they were made and expensive by the time anyone noticed. Each one gets the argument for it before the argument against.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Every one of these was chosen deliberately by a competent engineer under a real constraint. What changed between then and the day it became the reason nobody trusts the platform?
The engineer who inherits the platform and has to decide what to fix first, and the analyst who has been working around these for a year without knowing which of their workarounds are load-bearing. Neither is helped by a list that only scolds.
The unit here is the decision, not the dataset. Each entry is one choice made at one moment with one justification, and the interesting property of each is the gap between the conditions that justified it and the conditions it is running under now (Data Architecture Patterns).
Read a list of anti-patterns, recognise several in your own platform, and open tickets to fix them all. The recognition is genuine and the instinct is right — and taken literally it produces a quarter of migrations that nobody asked for while the two that actually hurt stay in place, because a list has no ordering and a platform has a budget.
The pattern is fixed and the constraint that produced it is not, so it comes back. A team that removes SELECT * from every model without giving anyone a schema they can rely on will find it reintroduced within a month, correctly, because the alternative was fragile (Data Contracts).
- The pattern is fixed and the constraint that produced it is not, so it comes back. A team that removes
SELECT *from every model without giving anyone a schema they can rely on will find it reintroduced within a month, correctly, because the alternative was fragile (Data Contracts). - The migration is more dangerous than the anti-pattern. Repartitioning a large table, consolidating twenty dashboard definitions, or moving analytics off the production database are all multi-week projects with their own failure modes, and doing them under the banner of cleanliness gets them funded badly and staffed thinly (What Backfills Break).
- The anti-pattern is load-bearing. Someone is reading the extract that overwrites source data, the tiny files are produced by the streaming ingest that a real consumer depends on, and the production-database query is the only thing keeping a team unblocked (Who Actually Consumes This Data).
- Two of them are the same problem. No schema ownership and business logic copied into twenty dashboards are both the absence of a contract boundary, and fixing either one separately produces half a solution twice (Data Ownership).
- The list is applied to a platform below the scale that makes any of it wrong. A three-person company querying its production replica is not committing an anti-pattern; it is correctly not building a data platform (OLTP vs OLAP).
What is actually happening
- Almost every entry follows one shape: a decision that was locally correct becomes globally wrong when a variable moves. The variable is usually volume, cardinality, consumer count or team count, and it moves slowly enough that no single day is the day it broke (Scaling from One User to Millions).
- The second shape is deferred cost. Skipping raw history, skipping lineage and skipping idempotency all make the first version ship sooner and make every subsequent repair harder. They are loans, and the repayment arrives during an incident rather than during a planning cycle (Keeping Raw History: The Recovery Position and the Liability).
- The third is absent feedback. Nothing in a platform reports that a dashboard has no freshness stamp, that nulls are being silently absorbed, or that a metric now has nine definitions. These persist not because anyone chose them but because nothing was ever going to say otherwise (Data Observability).
- Physical anti-patterns share a mechanism the others do not: they are decided once, at write time, and everything read afterwards inherits them. Partitioning, file size and column selection are the highest-leverage and least reversible decisions in an analytical platform, which is exactly the combination that produces regret (Physical Data Layout).
- Organisational anti-patterns are not technical debt at all. No schema ownership and twenty dashboard definitions are coordination failures, and every attempt to fix them with tooling alone reproduces them inside the new tool (Data Governance).
Anti-patterns of place: where the bytes actually sit
The four entries below share a property that makes them the most expensive on the list: they are decided at write time and paid at read time, by everyone, forever. Nothing about them is visible in a query, a test or a run report — they show up as a platform that is slowly getting slower for reasons nobody can attribute (Physical Data Layout).
They also share the most sympathetic origins. Querying the production database is how every company starts and is correct for longer than most architecture diagrams admit. Selecting every column is how you avoid breaking when an upstream schema changes under you. Partitioning by the identifier you filter on is a direct application of the advice one lesson earlier. Writing a file per micro-batch is what a streaming ingest does by default.
The layout below shows the sharpest of the four. A table partitioned by a per-user identifier is partitioned by something with as many distinct values as it has users, which means a directory per user, a tiny file inside each, and a predicate on any other column that can prune nothing at all. The engine does not fail; it lists every partition, opens every file and reads all of them, and every one of those steps is metadata work that a larger cluster does not accelerate (Partition Pruning, Partition Cardinality).
- orders/user_id=100001/a handful · 3 files · read
- orders/user_id=100002/a handful · 2 files · read
- orders/user_id=100003/a handful · 4 files · read
- orders/user_id=… (one directory per user)a handful each · 1 file · read
- orders/order_day=2026-03-18/ (the alternative)one day · 8 files · read
Both layouts hold the same rows and answer the same question. The difference is entirely in what the reader is allowed to skip, and it was fixed months earlier by whoever chose the partition key (The Partitioning Decision).
| Anti-pattern | Why it was tempting | What it actually costs | The cheapest way out |
|---|---|---|---|
| Using the production database as the data warehouse | It is already there, already correct, already fresh, and it has no second system to operate. For one product and a few analysts this is genuinely the right architecture, and every alternative is overhead (OLTP vs OLAP). | The first query that scans years of history competes with customer traffic, or lags the replica the application reads. The operational schema also holds current state only, so any question about the past is unanswerable rather than slow (Workload Isolation). | A replica first, then an extract into columnar storage for the historical questions specifically. Name the trigger — analytical load affecting production, or a second source appearing — rather than migrating on principle. |
SELECT * in models that other models read | It survives upstream additions without a code change, it is faster to write, and it makes an exploratory query into a model with no editing. In an ad-hoc query against a small table it remains fine. | Columnar storage exists so a reader can open only the columns it needs; selecting everything discards that on every downstream query. It also propagates every upstream schema change into every consumer, which is how a rename breaks eleven dashboards (Projection Pushdown). | Enumerate columns at the boundary where consumers start reading — the curated layer — and leave raw and staging permissive. The property you want is a stable contract, not universal verbosity (Model Layering). |
| Partitioning by a very high-cardinality key | The query filters on it, and partitioning by what you filter on is the entire idea. With a user id or an order id the reasoning is locally impeccable (Partitioning). | One directory per distinct value, one tiny file inside each, and a partition set expensive to list before any data is read. Predicates on anything else prune nothing, and the cost grows with cardinality rather than volume (Partition Cardinality). | Partition on a low-cardinality column — usually a date — and use clustering or sort order for the selective high-cardinality predicate, which prunes within files rather than between directories (Clustering and Sort Order). |
| Millions of tiny files | Every streaming ingest and every frequent micro-batch produces this by default, and each individual write is correct. Nothing raises an error, and the first thousand files behave fine (Streaming Ingestion). | Read planning becomes dominated by listing and opening objects rather than by reading rows, compression has too little to work with, and the job that used to finish inside its window stops doing so. A larger cluster does not help, because the work is metadata (File Size and the Small-Files Problem). | A compaction job that rewrites small files into larger ones on a schedule, and a write path that batches before landing. Both are ordinary maintenance and both need a window and an atomic swap (File Compaction, Atomic Publish). |
Read the second column before the third every time. An engineer who understands why each was tempting will recognise the temptation next quarter; one who has only memorised the third column will make a new version of the same mistake with different nouns.
Anti-patterns of time: history, re-runs and repair
The next four are about the future rather than the present. Each of them makes the current pipeline simpler and makes every subsequent repair harder, and the bill arrives at the worst possible moment — during an incident, when you discover which options you spent.
The unifying question is: when this is wrong, what can you rebuild it from? A platform that retains raw arrivals and has deterministic, bounded, re-runnable transformations can recover from almost any mistake by reprocessing. A platform that cleaned data on the way in, overwrote what it read, and has a pipeline that only runs forward has no recovery path at all, and its first serious bug is permanent (Reprocessing vs Retrying).
The fourth entry is different in kind and belongs here because it is the failure of the repair itself. A backfill is a large write against production data, competing for the same compute as live queries, touching partitions that consumers are reading. Running one without a capacity plan is how a correction becomes a second incident, and it happens because a backfill looks like a re-run and is treated like one (What Backfills Break).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| No raw history — data is cleaned, typed and filtered on the way in, and only the cleaned copy is retained. | A transformation bug is found six months later. The correction cannot be applied, because the input it needed was discarded during ingestion and the source has moved on. | Cleaning at ingestion looks like efficiency: less storage, fewer columns, one less layer. What it actually does is make the ingestion logic the one piece of code in the platform that can never be wrong (The Raw Landing Zone). | Land raw exactly as received, including fields you do not use, and clean in a separate layer. Storage is the cheapest thing in the chain and the discarded field is the one a question next quarter needs (Keeping Raw History: The Recovery Position and the Liability). |
| Transformations mutate the source data in place — updating a status, normalising a value, deleting rows judged invalid. | The pipeline and the source disagree and nobody can establish which was right, because the evidence was the thing that got overwritten. | It saves a copy and it is how you would fix a row by hand, so it feels like the same operation at scale. It also destroys the only artefact that could reconcile the two ends of the chain (Reconciliation). | Treat every input as immutable and write corrections as new rows or new partitions. Where the source genuinely must be corrected, that is an application change owned by the application team (Data Engineering and Backend Engineering). |
| A pipeline that cannot be re-run — it appends unconditionally, reads the clock, or depends on a dimension that has since changed. | Re-running to repair a period either duplicates it or produces a different answer than the first run, so nobody re-runs anything and errors accumulate instead of being fixed. | Idempotency is invisible until the first repair. An append is simpler than a merge, current_date is simpler than a parameter, and both work perfectly on the happy path for as long as the happy path lasts (Idempotent Data Pipelines). | Parameterise the period, merge or replace by business key rather than appending, and snapshot or version the dimensions a run depends on. This is the precondition for ever fixing anything (Upserts and Merges, SCD Type 2 in Practice). |
| A backfill of many months, run against production compute during business hours, writing into live partitions. | Dashboards slow to a crawl, the current period briefly contains historical data, and the repair produces a larger incident than the fault it was repairing. | A backfill reuses the same code as the nightly run, so it is mistaken for the same operation. It is not: it is a bulk write, over a range nobody has validated, against data consumers are actively reading (Planning a Backfill). | Bound the range explicitly, run on isolated compute, write to a staging location, validate against a period the bug never touched, then swap atomically. And plan the capacity, because the run is the largest job the platform executes that month (Validating a Backfill Before You Publish, Atomic Publish). |
Anti-patterns of meaning: the checks nobody wrote
The six entries here are invisible by construction. Nothing in a platform reports that a column is silently absorbing nulls, that duplicates are being counted, that a dashboard has no freshness stamp, or that the word "revenue" now has nine implementations. They persist because no mechanism was ever going to mention them (Data Observability).
The most useful way to teach them is as absent checks, which is what the table below does. Each row names the assertion whose absence defines the anti-pattern, what having it would express, what it catches, and — as always — what it would still miss. Read the last column carefully: several of these checks are commonly installed in a form too weak to catch the thing they were installed for (Data Tests).
Two of the six are organisational rather than technical, and their checks are correspondingly odd. "Every dataset has a named owner" is asserted against a catalog rather than against data, and "every published metric has exactly one implementation" is asserted against a registry. Both are still checks, both can fail a build, and treating them as documentation instead is how they stop existing (Data Ownership, The Metrics Layer).
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Every ingested dataset has a named producing owner and a schema they agreed to | Somebody is accountable for the shape and meaning of what arrives, and a change is a change to an agreement rather than a surprise. | Renames, retypes and removals arriving without notice; a field whose meaning changed with no schema diff at all; the situation where two teams each assume the other owns a table (Data Contracts, Semantic Changes). | Everything about correctness. A contract asserts that the shape is as agreed, never that the values are right, and an owner who agrees to a schema has not agreed to the data being accurate (Contract Enforcement). |
| Casts fail loudly; null rates per column are asserted against a threshold | A value that cannot be parsed is an error rather than an absence, and a column that starts emptying out says so. | A type change upstream that silently nulls a measure, a unit or format change, a join key that stopped matching. This is the fault class that preserves every row count and destroys every value (Nullability & Defaults). | A well-typed wrong value, and any null that was always there — the check is on the *rate changing*, so a column that has been half empty since inception passes indefinitely. |
| Uniqueness on the business key, plus a duplicate count tracked over time | One row of this dataset is one of the thing it claims to be, and inflation would be visible before it becomes a number. | At-least-once redelivery, non-idempotent re-runs, fan-out joins against dimensions with duplicate or overlapping keys (Duplicate Rows). | A duplicate that arrived under a new key — a producer retry with a fresh event id looks like a second genuine event, and this check will cheerfully confirm both keys are unique (Deduplication). |
| Column-level lineage generated from the transformation code that ran | Every field can be traced upstream to its source and downstream to everything that depends on it. | Blast radius before a change, and the cause during an incident. Without it, impact is estimated by asking around and incidents are searched rather than traced (Impact Analysis, Lineage Debugging). | Anything the tool cannot parse. Logic in stored procedures, notebooks, spreadsheets and the BI layer produces no edges, and a graph that stops early looks exactly like a simple pipeline (Column-Level Lineage). |
| Every published metric has exactly one implementation, and dashboards consume it | One word means one quantity across the organisation, with an owner and an effective date. | Definitional drift between teams, the twentieth copy of a filter set, and the specific incident where two executives receive two correct numbers for the same day (Two Dashboards, Two Numbers). | The case where every dashboard agrees and every dashboard is wrong, because they share one incorrect upstream model. Parity and correctness are independent properties (Reconciliation). |
| Every dashboard renders the publish time of the dataset behind it | The person about to act on a number can see how old it is without asking anyone. | Stalled pipelines, paused schedules, a mart refreshing slower than its parent, and the BI cache — which no server-side check can see (Freshness Monitoring). | Everything about correctness, and it depends entirely on the reader looking at it. It is a control rather than an alert, and it works because it is always on screen (Stale Dashboards). |
The pattern across all six: the anti-pattern is not a thing anyone did, it is a thing nobody did. That makes them cheap to fix and almost impossible to notice, which is the reverse of the physical entries and the reason they survive so long in otherwise well-run platforms.
Anti-patterns of choice: streaming everything, and starting from the vendor
The last two are architectural rather than local, and they are the two most likely to be defended in a meeting rather than discovered in a query. Both begin with a genuine argument. Streaming really is the right answer for some consumers, and a managed platform really does remove operational work that a small team cannot afford.
"Stream everything" fails not because streaming is bad but because it is bought without the requirement that justifies it. A stream introduces state, event-time reasoning, watermarks, lateness policy, replay semantics and a permanently harder debugging story, and every one of those is a real ongoing cost. When a consumer needs seconds, that cost is worth paying. When the dashboard is read once each morning, the platform has bought all of the cost and none of the benefit — and often ends up *less* fresh, because the incidents are harder and last longer (Batch vs Streaming Ingestion, Watermarks).
Vendor-first architecture fails by inverting the order of two questions. The right sequence is: what does the consumer need, what data shape does that imply, which primitive answers it, and then which product implements that primitive well for us. Starting from the product answers question four first and then works backwards, which reliably produces a platform whose components are correct individually and unjustified collectively (Data Architecture Patterns).
Neither of these is fixed by a rule. They are fixed by making the requirement explicit before the component: a written latency requirement per consumer, and a written statement of the problem each component was added to solve. Both are cheap, both are boring, and a platform with them does not accumulate either anti-pattern (Who Actually Consumes This Data).
A continuously running stateful job with checkpoints, lag and backpressure is a system you operate, where a batch job is a thing you re-run. This is the largest and least-budgeted line.
Windowed aggregates and stream joins hold state proportional to the window and the key cardinality, and that state must be checkpointed durably to survive a restart (Streaming State).
A stream holds capacity continuously, including overnight and at weekends, where a batch job holds it for the length of the run.
Reproducing a streaming bug requires replaying a specific window with a specific watermark position, where a batch bug is reproduced by re-running the job over a period (Replay from the Log).
The benefit, and the only entry on this chart that is a benefit. It is large — where a consumer has a genuine latency requirement, nothing else buys it.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for the comparison, not measurements of any platform. The point of the chart is that the benefit row is genuinely large, and that it is worth nothing at all for a consumer who reads the number once each morning (Cost vs Freshness).
Which consumer has a latency requirement, what decision does that latency serve, and what happens if the number is an hour old?
when Every consumer of this source reads it on a human cadence — a morning report, a weekly review, a monthly close. Which is most of them.
cost Freshness bounded by the interval, and a coarse interval means a period is either complete or absent. Cheap, restartable, debuggable, and re-runnable from raw (Batch Ingestion).
when A consumer wants minutes rather than seconds, and the source can be read incrementally by a reliable watermark or a change feed.
cost Watermark state, late-arrival handling and the possibility of a permanently skipped record if the watermark advances past uncommitted rows (Incremental Processing, The High-Water Mark).
when A consumer acts on the data within seconds and the action has value that decays that fast — fraud interdiction, operational alerting, in-product personalisation.
cost Durable state, event-time and watermark reasoning, a lateness policy, replay semantics, and an on-call story that is materially harder than a batch job that can simply be re-run (Stateful Stream Processing).
when One consumer genuinely needs seconds and another needs a reproducible closed period, and no single path serves both well.
cost Two implementations of the same logic that can disagree, which is the specific cost Lambda-style architectures pay and the specific reason unified engines are attractive (Lambda Architecture, Batch and Streaming Unification).
Managed streaming, warehouse and lakehouse products change their capability boundaries frequently, and the boundary is exactly what a vendor-first architecture depends on. Treat any specific claim about what a product now handles for you — serverless scaling, automatic compaction, managed schema evolution — as something to verify against current documentation rather than something to design around from memory.
How to build it
Most important first.
- Rank by blast radius and reversibility, not by how much each one offends you. A partition key that has to change is worse than a hundred
SELECT *queries, because one is a rewrite of history and the other is a series of small edits (The Partitioning Decision). - For each one you decide to keep, write down the condition that would make it wrong. "Querying the replica is fine until an analytical query causes replication lag that the application notices" is an operational trigger. "We should really move off this eventually" is not (Workload Isolation).
- Fix the mechanism that reintroduces the pattern before fixing the instances. Give consumers a stable curated model and
SELECT *stops being the rational choice; publish a metric definition and the twentieth copy stops being written (Model Layering, The Metrics Layer). - Buy the cheap irreversibles early. Raw retention, an event id on every record, and a bounded re-runnable pipeline cost almost nothing before there is data and cannot be added retroactively to data you no longer have (The Raw Landing Zone, Idempotent Data Pipelines).
- Make the invisible ones visible rather than fixing them outright. A freshness stamp on every dashboard, a null-rate panel per column, and a duplicate count per business key convert three silent anti-patterns into three numbers somebody can argue about (The Data Quality Dashboard).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- None of these breaks a guarantee the platform ever made. That is what makes them anti-patterns rather than bugs: every one of them produces a system that works, and the cost is paid in properties nobody wrote down.
- What they remove is optionality. Overwriting raw data removes the ability to reprocess; a high-cardinality partition key removes the ability to prune; no lineage removes the ability to establish blast radius. In each case the guarantee lost is one you did not know you were holding (Impact Analysis).
- Two of them actively create false guarantees. Silent null handling makes a column look populated, and ignoring duplicates makes an additive measure look larger, and both survive every check that does not specifically look for them (Nullability & Defaults).
- Fixing one does not guarantee the others improve. These compound but they do not chain, and a platform can have excellent layout and no ownership at all.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that finds most of this list is an inventory rather than a test: for every serving dataset, does it declare a grain, an owner, a freshness target and a retention rule, and is there a check asserting each? The gaps in that table are the anti-patterns you actually have (Dataset Documentation).
- It misses everything physical. No inventory notices a partition key with impossible cardinality or a directory holding millions of files — those are found by looking at the storage layer directly, which nothing does automatically (Partition Cardinality).
- It also misses the organisational entries entirely. No query can tell you that two teams believe they own the same schema, or that neither does (Who Owns Data Quality).
- Several of these degrade freshness without changing any schedule. Millions of tiny files make every read slower until a job that used to finish inside its window does not; a full refresh that rebuilds all history takes longer every day by construction (File Size and the Small-Files Problem, Full Refresh vs Incremental).
- Dashboards without a freshness stamp do not change freshness at all — they remove the consumer's ability to observe it, which is worse, because a stale number that announces itself is merely late and a stale number that does not is wrong (Stale Dashboards).
- "Stream everything" usually improves the freshness nobody asked for and degrades the freshness people rely on, because streaming state, ordering and lateness handling are new sources of delay and of incidents (Batch vs Streaming Ingestion).
- The anti-patterns that hurt most on a schema change are the ones with no boundary:
SELECT *propagates every upstream change to every consumer, and no schema ownership means the change arrives without notice (Schema Evolution, Breaking Schema Changes). - Silent null handling is a schema-evolution anti-pattern wearing a transformation costume. A cast that yields null rather than raising converts every future type change into a silent data change (Contract Enforcement).
- Physical anti-patterns are stable under schema change and terrible under scale change. They are the ones you can leave alone through a rename and cannot leave alone through a large customer onboarding (Data Skew).
- The recoverable ones are the ones that only affect what happens next:
SELECT *, silent nulls, missing freshness stamps, copied metric logic. Fix forward, and no history has to move. - The expensive ones require rewriting data: partition key changes, file compaction, and any correction that has to be applied to periods already published. Bound them, run them off the serving path, and validate against a period the fault never touched (Planning a Backfill, File Compaction).
- One of them has no recovery at all. Data that was overwritten in place, or never retained in raw, cannot be recomputed at any price — which is why it is the one item on this list worth fixing before it applies to you (Keeping Raw History: The Recovery Position and the Liability).
What can go wrong
- The list is used as an audit and produces a document, which is filed. Nothing about the platform changes and everyone now believes the problem is understood.
- A migration is launched to fix a physical anti-pattern and is abandoned half-done, leaving the platform with two layouts, two sets of queries, and nobody sure which is authoritative.
- The team fixes the visible instances and not the mechanism, so the pattern returns and is now accompanied by a rule everyone routes around.
- A pattern is removed that was genuinely load-bearing, and a consumer discovers it in production (Impact Analysis).
- The organisational entries are assigned to the data team, which has no authority to name owners, and are reported as blocked for a year (Data Ownership).
- "We have none of these, our platform is fine." Every platform has several. The useful question is which ones are currently costing you something and which are dormant because a variable has not moved yet.
- "
SELECT *is fine in analytics." It is fine in an ad-hoc query against a small table. In a model that other models read, it defeats projection pushdown on every downstream query and propagates every upstream schema change to every consumer (Projection Pushdown). - "Partition by every commonly-queried column." Each partition column multiplies the partition count, and a partition set large enough to be expensive to list costs more than the pruning saves. Partitioning is a physical decision with a cardinality budget (Partitioning).
- "Streaming is more modern, so we should stream everything." Streaming buys freshness and costs state, ordering, lateness handling and a permanently harder debugging story. Buy it where a consumer has a latency requirement and nowhere else (Batch vs Streaming Ingestion).
- "The vendor's reference architecture cannot be an anti-pattern." A reference architecture is an answer to a generic problem, and the components in it are the ones the vendor sells. It is a reasonable input and a poor requirement (Data Architecture Patterns).
- "These are junior mistakes." Every entry on this list is most often introduced by an experienced engineer moving quickly under a real constraint, which is why they survive review and why the constraint has to be part of the explanation.
- Two entries are compliance exposures rather than engineering ones. Overwriting source data destroys the audit trail that a regulator would ask for, and no lineage means a deletion request cannot be traced to every copy it must reach (Deletion Requests, Data Lineage).
- Keeping raw history forever is the strongest recovery position and the largest privacy liability on the platform. Retention is where those two arguments meet, and neither side wins outright — which is why it belongs in a written policy rather than in a storage-cost conversation (Data Retention).
Operating it
- A per-dataset inventory: grain, owner, freshness target, retention, tests. Missing cells are the finding (The Data Catalog).
- File counts and average file size per partition path, which is the only way any of the physical entries becomes visible before it becomes slow (File Size and the Small-Files Problem).
- Distinct partition count per table over time. A partition count growing proportionally with rows is a cardinality mistake announcing itself (Partition Cardinality).
- The number of distinct implementations of each published metric, and the number of dashboards bypassing the definition layer (The Metrics Layer).
- Null rate and duplicate count per business key, per dataset, tracked rather than alerted, because both are drifts rather than events (The Dimensions of Data Quality).
- At 10x volume the physical entries stop being tolerable: a query that scanned an acceptable amount now scans ten times that, and the tiny-file problem becomes a planning problem rather than a read problem.
- At 10x cardinality, partitioning choices invert. A key that produced a sensible number of partitions produces an unusable number, and the fix is a rewrite of the table (Partition Cardinality).
- At 10x consumers, the organisational entries dominate everything else. Twenty dashboards with copied logic is an annoyance; two hundred is a platform whose numbers cannot be reconciled at all (Who Actually Consumes This Data).
- Below the scale that forces any of this, most entries on the list are not mistakes. The production database as the analytical store is the correct architecture for a company with one product and three analysts, and saying so is part of teaching the list honestly (OLTP vs OLAP).
- The physical entries drive bytes scanned more than anything else in a platform. Selecting every column defeats projection pushdown, a partition key that cannot prune defeats partition pruning, and both are paid on every query by every consumer forever (Scan Cost, Projection Pushdown).
- Tiny files drive a cost that does not look like data at all: listing, opening and planning over an enormous number of objects, which is metadata work rather than scan work and is not fixed by a bigger cluster (File Compaction).
- Full refreshes drive work repeated: recomputing all of history to add one day, with cost growing linearly in history rather than in new data (Compute Waste).
- The organisational entries drive engineering hours rather than platform cost, and those hours are the most expensive line in any data platform. Reconciling nine definitions of one metric each quarter is a recurring cost nobody bills to the decision that created it (Cost Attribution).
- Every fix here costs something real. Curated models cost a layer to maintain; contracts cost producer friction; compaction costs a job and a window; a metrics layer costs a component and an argument. A list of anti-patterns that does not price its own remedies produces a platform with sixteen new obligations.
- Fixing the reversible ones first feels like progress and delays the irreversible ones, which are the only ones that get worse while you wait. Retention and layout should jump the queue precisely because they are unpleasant.
- Making an invisible anti-pattern visible does not fix it and does add an uncomfortable number to a dashboard. That discomfort is the mechanism by which it eventually gets fixed, and it will be unpopular in the meantime.
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe mechanism behind each entry — a locally correct decision meeting a variable that moved — is independent of stack. Which entries bite first depends heavily on the platform: warehouse-centric shops meet the scan and definition entries first, lake-centric shops meet file size and partitioning first.
- SCALE-SPECIFICQuerying the production database, skipping a metrics layer and running full refreshes are all defensible below a certain volume and consumer count, and become expensive above it. The list is not a set of rules; it is a set of thresholds, and naming your own threshold is the actual exercise.
- ORG-SPECIFICSchema ownership, duplicated metric logic and vendor-first architecture are coordination outcomes rather than technical ones. In a single-team platform none of the three can occur; above roughly three independent producing teams all three occur by default unless something prevents them.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns the guarantees that "stream everything" quietly commits you to — delivery semantics, ordering across partitions, and what a replay actually replays after a restart.
- — DevOps / Production Engineering owns the change-management half of several entries here: how a schema change is reviewed and promoted, how a backfill is planned as a production change rather than a re-run, and why a rule that can be skipped under deadline pressure will be.