Learn Data Engineering

Not the tools. The discipline of moving data between systems so that what arrives is complete, correct, explainable and affordable.

Application to PostgreSQL to extract to lake to transformation to warehouse to mart to dashboard — and what each arrow is actually promising.

Thirteen failure classes, each with its own mechanism, its own detector, and a long list of checks that will never find it.

Databases own storage, distributed systems own guarantees, backends own the transactional service, analytics and ML are consumers. We own movement, transformation, modelling, validation and serving.

Source, ingestion, raw, transformation, validation, storage model, serving, consumer, feedback — nine stages that are a design order forwards and a diagnostic order backwards.

One authoritative system per business concept, everything else explicitly a copy — and the discipline that follows once you have said which is which.

Analysts, finance, product analytics, ML, agents and operational read-back each need a different freshness, tolerate a different amount of revision, and fail in a different way. Design from them inward.

Trust is not a feeling about a dashboard — it is a set of questions a consumer can answer without asking you. It is built slowly, lost in one incident, and rebuilt at a much higher price.

Many small transactions against current state: point lookups by key, a few rows written, low latency, high concurrency. The shape that explains every design choice an operational database makes.

Few queries, each reading a large range of history and collapsing it into a handful of numbers. Judged on throughput rather than latency, bound by bytes moved rather than by seeks, and unhelped by almost every index you could add.

The two workloads compared on every axis that actually differs — plus an honest account of where the line has blurred, where it has not, and why the distinction still decides your architecture.

What an analytical query actually takes from the operational system it runs on — and the specific, honest cases where running it there is still the right call.

Production database, extraction, transport, raw landing, transformation, analytical storage, BI query — what each hop buys, what it actually promises, and where the shape of the data changes underneath you.

The same four columns of the same table, written to disk two ways — and exactly which bytes each query is then obliged to move as a result.

What an engine actually does with `SELECT avg(spend) FROM users WHERE country = 'DE'` once the data is stored by column: chunks opened, blocks skipped, batches decoded, a predicate evaluated into a mask, and one running total.

Columns of one type with repeating values compress in ways rows of mixed types cannot — and the chain from fewer bytes to a faster query has three places it can break.

Four type-aware encodings, what redundancy each one exploits, and the column property — cardinality, sortedness, range — that decides whether it does anything at all.

A self-describing, columnar, splittable file whose footer tells a reader what it can skip — which is a different claim from "it is smaller".

Row groups, column chunks, pages and the footer — where statistics come from, why they are only useful when the data is sorted, and how nesting is stored without abandoning columns.

Follow `SELECT country, revenue FROM events WHERE date = '2026-08-25'` from a directory listing to decoded values, and count what got skipped at each of the four gates.

Row-oriented binary records with the schema travelling alongside the data — built for exchange and evolution rather than for scanning one column across a billion rows.

A sibling columnar design with the same goals and different specifics: stripes instead of row groups, row-index strides for finer skipping, and row-level ACID in the ecosystem it grew up in.

Not a rivalry. One is built for reading a few columns across many rows, the other for handling whole records one at a time — and most pipelines use both, in that order.

No types, no schema, no statistics, ambiguous quoting and — for CSV — a splittability problem that has no clean fix. And still the right answer for interchange, small data and human inspection.

Moving data out of systems you do not control and into storage you do — and the difference between what arrived and what happened.

Databases, APIs, logs, files, event streams, SaaS systems and object storage — seven extraction models, seven failure behaviours, and seven different meanings of "everything since last time".

Every hour, select what is new, write files, load the warehouse. The simplest thing that works — and the specific ways it stops working.

Asking a source for "everything since last time" — and the specific, silent, permanent ways `WHERE updated_at > :last_run` gets that wrong.

Event happens, producer publishes, broker durably holds, consumer reads, storage lands it. Continuous rather than windowed — and continuously running.

Not old versus new. Two designs with different freshness shapes, different failure surfaces, different recovery stories and very different operational burdens.

What to do when an extract fails, a connector stalls or a consumer falls behind — and how to tell, quickly, whether the data is late or gone.

Land what arrived, exactly as it arrived, including the fields you do not use. Never clean in place. Partition by arrival so a re-run is bounded.

Extract, transform, load. The destination only ever sees rows that already conform — a real guarantee for its readers, bought with the original that nobody kept.

Extract, load, transform. The destination holds the raw copy and does the work — which turns most transformation bugs into a re-run and hands the destination every obligation the raw data carries.

Two orderings, six criteria. Where compute is available, how large raw is, whether you need history, what security forbids, how complex the logic is, and what the destination can actually do.

In the source, in a dedicated cluster, in the warehouse, in the query at read time, or in the BI tool. Each placement moves cost, freshness, testability and governance somewhere different.

Three jobs that need separating — preserve what arrived, make it usable, model it for consumers. The names vary by house; the purposes do not.

Bronze, silver, gold is a widely used set of names for raw, staging and curated. It is a convention, not a requirement, and it is not an architecture.

An immutable copy of what arrived is what makes every downstream mistake fixable. It is also the most sensitive dataset the platform holds. Both are true and neither cancels the other.

One place to land structured, semi-structured and unstructured data before anyone knows which questions it will answer — and the reason most lakes become swamps.

An analytical database built for large scans and aggregations over structured, modelled data — and what it gives you that a pile of files cannot.

Object storage for the bytes, a table metadata layer for the transactions, and independent query engines on top — a combination rather than a product.

How Iceberg, Delta and Hudi turn immutable objects into a transactional table: a manifest of which files are the table now, and a commit that is one pointer swap.

A comparison across data types, query patterns, governance, cost shape, openness, transactions and tooling — with the vendor framing removed and the overlap admitted.

Buckets, keys, objects and metadata — and the four properties of that model that decide how every data pipeline above it must be written.

A narrow, purpose-built, usually pre-aggregated serving copy that trades flexibility and freshness for query cost and simplicity.

Scale each independently, point many engines at one copy, pay for compute only while it runs — and pay a network, a cold start and the loss of locality for it.

Two datasets with identical rows and identical schemas can differ by an order of magnitude in what a query must read. The difference is which rows share a file and which files share a directory.

The same bytes split into a million objects behave nothing like the same bytes in a hundred. On object storage the cost is per request, so this is a problem about file count, not data volume.

Rewriting many small files into fewer larger ones. What it buys, what it costs, and what a reader sees if it is halfway through when their query starts.

Putting a column's value into the directory path so a reader can exclude data without opening it. The cheapest skip available, and the only one that costs nothing to evaluate.

The planner eliminating partitions before reading. It is a best-effort behaviour, not a guarantee — and there are four common predicate shapes that silently defeat it while looking completely correct.

A partition key with too many distinct values produces more metadata than data. The target is enough rows per partition to justify a file, and few enough partitions to list.

Within a partition, the order rows were written in decides how selective file statistics are. Sorting is what turns a min/max into a skip, and it decays as soon as you stop maintaining it.

Hashing a key into a fixed number of files so that rows with the same key always land in the same bucket. One physical technique, and it earns its keep almost exclusively when it lets a join skip the shuffle.

Five questions that decide a partition key: what the filters carry, how many distinct values, how much data per partition, how often data is appended, and whether the distribution is skewed. Answer them from data, not from intuition.

Choosing the shape of the tables people query, so that the questions the business asks are easy to write, correct by construction and affordable to run.

Normalised, transaction-oriented schemas and fact/dimension, query-oriented schemas solve different problems. Neither is a degraded version of the other.

Tables of measurements of a business process, at a declared grain, with keys to context — and the measure types that decide whether SUM() means anything.

The single most important question in analytical modelling. Answer it in one sentence per table, or every aggregate downstream is a guess.

The descriptive context you filter and group by — customer, product, date, region — and the reason a calendar deserves a table of its own.

A warehouse-generated key with no business meaning, because with history the natural key stops being unique and because source ids change underneath you.

One fact table in the middle, dimensions one join away on every side. The shape that makes queries short, joins predictable and grain visible.

Dimensions normalised into their own hierarchies. Fewer repeated values, one place to correct a taxonomy, more joins in every query — and an honest comparison of when that trade pays.

A customer moves from Poland to Germany. Do last quarter's Polish revenue figures change? That question, answered per attribute, is the whole topic.

valid_from, valid_to, is_current — the columns that preserve history, the join predicate every fact must use, and the interval bugs that produce numbers reconciling against nothing.

Capture the state of every entity at the end of every period. `account_balance_daily` answers "what was it on the 14th" with a lookup instead of a fold over all history.

Events record what changed; snapshots record what was true at time T. Different storage curves, different query complexity, and different questions made easy.

Clean, cast, join, aggregate, deduplicate, enrich, filter, normalize, denormalize — nine operations, each with a way of being wrong that does not raise an error.

The five-line aggregate everyone writes, and the thirty-line one that is still correct after duplicates, refunds, currency and late data exist.

Transformations as version-controlled, tested, documented models whose dependencies are inferred rather than declared — and materialisation as a choice you make on purpose.

raw_orders to stg_orders to int_orders_enriched to fct_orders to customer_metrics — five nodes, four edges, and everything you can ask of a graph you did not have to draw.

Node, edge, no cycles — the whole structure. Why every question a data platform asks about itself turns out to be a standard graph traversal, and why a cycle is almost always a modelling error.

If A feeds C and B feeds C, then A and B can run together and C must wait. What that ordering buys — parallelism, correctness of order, selective rebuild — and the one thing it emphatically does not buy.

Staging renames and types one source. Intermediate joins and reshapes. Marts face the business. The rule that makes it work is that consumers depend only on marts.

Business logic copied into twenty dashboards produces twenty definitions of revenue, all defensible. A metric defined once, with an owner, is the only fix — and it does not fix everything.

Coordinating work by dependency, state and time — deciding not only when a task may start but whether it should, and what its result means.

"Run at 02:00" versus "run B after A succeeds, retry C, skip D if the source is empty" — a difference in what the system remembers, not in how it is configured.

A DAG of tasks, a scheduler, a metadata database and workers — and the logical data interval, which is the most misunderstood idea in orchestration.

The edges are the program. What "B runs after A" means when A is skipped, when A is upstream of forty tasks, and when B secretly reads a table nobody declared.

A succeeded, B succeeded, C failed. Re-run everything, only C, or C and everything downstream? The answer is decided entirely by idempotency.

Re-running the same logical input must not corrupt or duplicate the result. A pipeline that cannot be re-run is a pipeline whose every bug is permanent.

Process what is new instead of recomputing ten years — and inherit, in exchange, every problem of state: watermarks, late data and two eras in one table.

"Processed through offset X" — the one piece of state that decides what a restart re-reads, and why a log position is a promise and a timestamp is a guess.

Reading committed changes out of the database's own transaction log, so downstream systems learn what happened instead of repeatedly asking what is true now.

One asks the database what is true now, on a loop. The other observes what the database committed. The difference is not speed — it is which changes are structurally invisible.

An operation, a before image, an after image and source metadata. Which of those you actually receive is decided by the source's configuration, not by CDC.

The source log has a total order and a transaction boundary. Publishing splits both, and every consumer that joins two tables inherits the consequences.

CDC starts from now. Everything that existed before now has to be read separately and stitched to the stream without a gap and without a duplicate that a later ordering guard cannot resolve.

A consumer falls behind. Whether that is an inconvenience or an unrecoverable data loss is decided entirely by whether the connector's position is still inside the source's retained log.

A migration runs on the source at 02:00. Some connectors emit a schema-change event, some silently reshape the payload, some stop. None of them ask you first.

An append-only, immutable, ordered sequence of facts that each reader moves through at its own pace — the primitive underneath brokers, replication, CDC and stream processing.

Two different products wearing one word. A queue distributes work and forgets; a log stores records and lets anyone re-read them. Neither is the upgrade of the other.

A partitioned, durable, append-only log with independent consumer groups and time-based retention. Records survive being read, which is the property everything else in a data platform is built on.

A topic is not one log — it is several. Ordering holds inside a partition and nowhere else, and that single sentence explains most of the surprises in a streaming platform.

The key hashes to a partition, and the partition is the scope of ordering. Change the partition count and the hash re-maps, so a key's future loses order against its own past.

A group divides a topic's partitions among its instances, one partition to at most one instance. Partition count is therefore the hard ceiling on parallelism, and instances beyond it do nothing at all.

Retention is not a storage setting. It is the maximum age of a bug you can fix by replaying instead of reconstructing — a recovery-window decision that happens to be paid for in disk.

Commit before processing and you get at-most-once. Commit after and you get at-least-once. There is no third option unless the commit and the output write share a transaction.

Computation over an input that has no end, where every result is provisional, time becomes a data field, and the job is a long-lived process holding state rather than a script that finishes.

Filter, map, transform: operators whose output for a record depends only on that record. The cheapest, most restartable, most rescalable thing a stream can do — and a much narrower category than it first appears.

Counting, joining, windowing and deduplicating all require remembering something between records — which turns a job into a database you have to operate.

Where the state physically lives, what makes it grow, how it is snapshotted and restored — and why state size, not throughput, is the number that decides whether a streaming job can be operated.

The time the thing actually happened, carried in the record itself — the only clock that makes a result reproducible when you process the same data again next year.

The wall clock of the machine doing the work: always available, always monotonic, never late — and the reason a replay produces a different answer than the original run.

The moment a record entered the processing system, stamped by one clock the platform controls — the timestamp that makes lag measurable and replay stable without pretending to know when anything happened.

An event happened at 10:00 and arrived at 10:07. The 10:00–10:05 window was already emitted. What happens next is a policy decision, and most platforms have made it by accident.

A window is a rule that turns an infinite stream into a set of finite groups you are allowed to aggregate — and choosing the rule decides your state size, your latency and what questions you can answer.

Fixed size, contiguous, non-overlapping: every event lands in exactly one. The cheapest window and the only family whose results you are allowed to add together.

Overlapping windows of fixed size, advancing by a smaller step. Every event lands in size ÷ slide of them, which is exactly the factor by which state, output and the risk of double counting all multiply.

Windows whose boundaries the data draws: a session runs until a key goes quiet for longer than a gap. Per-key, data-dependent, mergeable — and the only window family with no upper bound on its own size.

An estimate of how far event time has progressed, derived from the data itself. It decides when a window may be emitted and what counts as late — and it is a claim, not a measurement.

Joining two unbounded inputs means holding both sides in state until a time bound says you may stop holding them. Without that bound it is not a join — it is a memory leak with a schema.

There is no single exactly-once guarantee — there are three separate questions, one about input consumption, one about state update and one about output write, and each is bought by a different, nameable assumption.

Splitting one computation across many machines, and the three things that buys you — memory, disk bandwidth and cores — against the one thing it costs: a network in the middle of your query.

A driver that plans and schedules, executors that hold data and run tasks, and a cluster manager that hands out machines. Almost every confusing Spark failure is explained by knowing which of the three it happened on.

A partition is the slice of rows one task processes alone. Too few and the cluster idles; too many and the scheduler dominates. And it is not the same thing as the partition in your storage path.

A stage is everything that can be done without moving data between machines. The boundary between two stages is always a shuffle, and it is always a barrier.

The one operation in a distributed job that uses the network for data. Every row is assigned a destination by key, written to local disk, fetched across the cluster and merged — which is why it dominates the runtime, the cost and the failure modes of almost every job.

Narrow: each output partition depends on one input partition, so the work stays where it is. Wide: it depends on many, so the data must move. This single distinction predicts every stage boundary in your job.

Real key distributions are not uniform. When one value holds most of the rows, the partitioner faithfully sends them all to one task — and that task becomes the job.

A job finishes when its slowest task does. One task out of a thousand taking twenty times as long makes the whole stage a twenty-times job, and no amount of extra capacity changes it.

Split the dominant key into several artificial sub-keys so its rows land in several partitions, then combine the partials. It works, it costs an extra stage — and applied to every key instead of the hot one, it does nothing at all.

When one side of a join is small enough to send everywhere, the large side never moves and the shuffle disappears. The whole technique rests on a size estimate — and on what happens when that estimate is wrong.

Transformations build a plan; nothing runs until an action asks for a result. That is what lets the optimiser see the whole query — and why your error message points at the wrong line and your pipeline ran three times.

The same question has many correct executions with wildly different costs. An optimiser turns what you asked into how it will run — using rules it can always apply and statistics it can only sometimes trust.

A stream-first distributed processor: a dataflow graph deployed once, records flowing through stateful operators, with checkpoints instead of re-runs. Compared with batch and micro-batch on what each one makes easy — not on which is better.

Engines that answer SQL over storage they do not own — what that separation buys, and every guarantee it quietly hands back.

Coordinator to workers to sources to partial results to merge — and the four places a query dies that a single-node engine never has.

Push the filter down to the reader so less is read at all — and learn the identical-looking query where it silently does not happen.

Read only the columns the query needs. The cheapest optimisation a columnar format offers, and the one `SELECT *` throws away.

If the remote system can filter, aggregate or limit, do the work near the data and move less — and know exactly which of those your connector actually supports.

One SQL statement across several systems. Genuinely useful, and it gives up consistency, predictable latency, optimiser competence and control of the load you impose.

Operators that process a batch of column values per call instead of one row at a time — and why that changes what the CPU is able to do.

Modern engines let you express both with one API. The authoring surface converged; latency, state and completeness semantics did not.

Every task green, every table populated, and the number still wrong. What "correct enough to trust" means, and why no single check establishes it.

Completeness, accuracy, freshness, uniqueness, validity, consistency — defined precisely, with how each is measured and which one is almost always asserted instead.

Assertions over rows and columns — not null, unique, non-negative, in a set, references valid — and the precise blind spot each one carries.

Every row is valid, every type is right, every key is unique — and today holds a small fraction of a normal day. The checks that compare data with its own history.

Expected latest data versus actual latest data — the cheapest check in the toolkit, two clocks that get confused, and the days it fires for no reason.

Count the rows and sum the measure at the source for a closed period, and compare with the serving table. The only check that observes both ends at once.

An alert nobody acts on trains people to ignore alerts. Severity from consumer impact, routing to the owner, and the difference between blocking a publish and sending a message.

One row per dataset — pipeline, freshness, completeness, status — and a hard rule that a green row is a statement about the checks you wrote, not about the data.

The team that produces a field owns whether it is correct. A data team can measure and report. Placing the whole obligation downstream guarantees it fails.

An explicit, owned, enforced agreement between the team that produces data and the teams that depend on it — covering names, types, nullability, meaning, freshness and how it may change.

Schemas change constantly. Adding, removing, renaming and retyping a field are four different risks with four different blast radii — and only one of them is routinely safe.

The producer ships first: data written under the new schema must still be readable, and still correct, for consumers that have not upgraded. State it as writer-and-reader, because the word itself is used in opposite directions by different communities.

The consumer ships first: data written under the old schema must still be readable, and still correctly interpreted, by code running the new one. In analytics this is not an edge case — every query over history is this question.

A shared, versioned store of schemas with a compatibility gate in front of it. It makes structural evolution mechanical — and it has nothing to say about meaning, ownership or the consumers reading your data by some other path.

A numeric field starts arriving as a string. Some consumers error; the dangerous ones cast, get null, keep every row, and report zero — with completeness, uniqueness and freshness all green.

The schema is identical, every type checks, every test passes, and the number now means something else. `revenue` went from gross to net. No tool will ever detect this — only documented semantics, an owner and a changelog will.

Making a field nullable is a breaking change for everyone who assumed it was not. A default hides missing data behind a plausible value. And "unknown", "not applicable" and "the pipeline dropped it" are three different facts stored identically.

Where the check actually runs — producer CI, the ingestion boundary, the entry to transformation — and the trade every enforcement point makes: a silent wrong number becomes a loud failure, which is correct and will still page someone.

Schema, owner, description, freshness, lineage, tags and quality — split by where each comes from, because that is what predicts which of them is still true.

Four questions it must answer, and the honest failure mode: a catalog nobody populates is worse than no catalog, because it looks authoritative.

orders DB to stg_orders to fct_orders to revenue_daily to the executive dashboard — and why that graph is a debugging tool rather than documentation.

orders.amount to revenue to monthly_revenue. Much harder to produce than table-level lineage, and the only granularity that answers the question an incident actually asks.

The same graph read the other way. If I change this column, what breaks — answered before the change rather than discovered afterwards.

Every important dataset has a clear owner. The failure to design against is "nobody knows where this table came from" — and it is an organisational problem with a technical trigger.

How someone finds the right dataset among hundreds. Search over descriptions fails; search over what people actually query works.

Documentation generated from the transformation graph stays true. Documentation written separately does not — and the distinction decides what is worth writing down at all.

Ownership, classification, access, retention and auditability as five mechanisms with enforcement points — not as a document in a wiki.

Public, internal, confidential, personal, highly sensitive — what the tiers mean, why the unit is the column, and why classification is worthless unless it propagates.

Where personal data actually ends up in a data platform: raw landing zones, debug logs, error messages carrying rows, notebook extracts, training sets, and the temporary table nobody deleted.

Store only what is needed — against the equally correct rule that says keep everything because you cannot recreate it. Both are right, and the resolution is structural.

How long should this dataset exist? A retention horizon is simultaneously a recovery window and a liability window, and the two want opposite numbers.

Least privilege applied to five surfaces — warehouse, lake, catalog, pipelines and secrets — where the weakest one is the effective policy and the pipeline is the most over-privileged principal you have.

An analyst sees EU rows only; a column comes back masked. Where that policy is evaluated decides whether it is a control or a convention — and a row filter silently changes what an aggregate means.

Four different techniques that people call masking. Which joins survive, who can reverse it, and why hashing a low-cardinality field is reversible by anyone with a loop.

A person asks to be erased from a platform built on immutable files, replayable logs and forty copies — and the backfill you run next week can bring them back.

Pipeline health and data health are two different systems. A platform that watches only the first finds almost none of the incidents anyone cares about.

What an orchestrator genuinely knows, what it structurally cannot know, and how to make a task-level signal say something about data.

Rows processed, bytes processed, duration, failures, retries and lag — what each one detects, what moves it for boring reasons, and what none of them can see.

Freshness is a per-dataset property. Averaging it across a platform hides the one table that has not updated since Friday — and the false-positive rate decides whether anyone still reads the alert in six months.

Comparing today with the same weekday historically is the cheapest broad detector there is — and it misses every error that preserves row count, which is most value-level bugs.

A dashboard says revenue dropped eighty percent overnight. Seven different causes produce that symptom, and telling them apart is the job.

Consumer symptom to serving dataset to transformation to upstream dataset to ingestion to source. Debug upstream, always — and diagnose from the set of checks that failed, not from the first one.

Click a dashboard metric and walk it back — tile, metric definition, mart, model, staging, raw, change capture, production database — then turn around and ask what else this feeds.

Recomputing history after the logic or the inputs changed — and why the hard part is publishing the result, not computing it.

Duplicated periods, overwritten current data, a saturated warehouse and a source knocked over by its own history — the four ways a correction becomes an incident.

Five questions to answer before the first partition runs: which range, is the re-run safe, where does the compute go, how do we validate, and how do we publish.

Reconcile the range against the source, explain every old-versus-new difference, and prove a period the bug never touched is unchanged — the check people skip.

The same button means two different things: finishing work that never completed, and redoing work that completed and is now wrong.

An event that happened on Tuesday and arrived on Thursday, after Tuesday was already computed, published and read.

Which key you deduplicate on decides which duplicates you can see — and a producer retry with a fresh id is invisible to every id-based scheme.

Replacing rows by key instead of appending them — the write that makes re-running safe, and the assumptions it quietly depends on.

Rebuild everything every time, or process only what changed. The first is expensive and has no state to get wrong, and it is the right answer more often than people admit.

Re-reading a retained event log versus recomputing from the raw layer — two recovery paths with different windows, different guarantees, and retention as the hard boundary on both.

Reliability is not a low failure rate. It is seven mechanisms — retries, idempotency, checkpoints, atomic publish, validation, rollback, reprocessing — that are only safe as a set.

A consumer must never read a half-written dataset. Build somewhere they are not looking, validate it there, then make it visible in one operation — and know which of the available operations is genuinely one.

A restarted job has to resume from somewhere. A checkpoint is correct only if it records the input position and the computed state together, in one atomic action — otherwise it is a dual write wearing a reliability hat.

Ninety-eight partitions succeeded and two failed. Re-running only the two is right — but only if the unit is idempotent and independently publishable, and most people check neither before doing it.

Retrying a task that already published is not a retry — it is a second publish. Retry is safe exactly when the task is idempotent, and a uniform retry policy applied to tasks that are not uniformly idempotent is the honest failure here.

A published, measured promise about a dataset — when it arrives, how fresh it is, how often it is right — agreed with the people who depend on it rather than declared by the team that runs it.

Now minus the event time of the newest **complete** data. The word "complete" does all the work: a table holding a few hours of today is extremely fresh and completely wrong to aggregate.

Reverting the transformation code does not revert the tables it wrote. A data rollback is either a restore from a retained snapshot or a re-run of the previous logic over the affected range — and both of them are forward operations.

Storage, scans, shuffle, compute hours, network, retention, file count and repeated work — put in the order they actually move the number.

What a query actually has to read, and why column selection and partition pruning are the two cheapest fixes in the entire domain.

Rebuilding history that did not change, refreshing models nobody reads, holding capacity nobody uses, and shuffling data that did not need to move.

Hot to warm to cold to archive to deleted, driven by how the data is actually read — and the retrieval cost that makes archive a trap for anything you might read again.

You cannot manage what you cannot attribute — and in a shared platform every cost belongs to everyone, which means it belongs to nobody.

The most direct trade in the platform: every increment of freshness is bought with compute that runs more often, longer, or continuously. The right freshness is set by the decision the data drives, never by what the stack can achieve.

Central warehouse, event-driven platform, Lambda, Kappa and mesh — sorted onto the two independent axes they actually live on, and compared by the problem each was a response to.

The arrangement most organisations actually run, taken seriously: one team, one place, one definition — with a real advantage and a specific failure mode that arrives with source count rather than with data volume.

Everything publishes events; consumers subscribe independently. It buys decoupling, replay and many materialisations of one stream — and it moves duplicate, ordering and schema handling from one place into every consumer.

A batch layer that is authoritative but late, a speed layer that is fresh but provisional, and a serving layer that merges them — bought with two implementations of the same logic that must agree forever.

One event log, one stream processing path, and reprocessing by replay. It removes Lambda's duplicated implementation and replaces it with two demands: the log must retain everything you might reprocess, and the stream job must replay history at a rate batch used to manage.

An organisational model, not an architecture: domain ownership, data as a product, a self-service platform and federated governance — with the operational cost of each stated honestly.

Owner, schema, semantics, quality, documentation, SLO, access policy. Seven commitments, and what a team has to start doing on the day it makes them.

Eight shared capabilities — ingestion, storage, compute, orchestration, catalog, quality, security, observability — and the boundary question that decides whether the platform team is a substrate or a queue.

An engineer declares a source and a model; the platform produces a pipeline, tests and monitoring. Get it wrong in one direction and it is a ticket queue with extra steps; get it right and you have four hundred datasets nobody owns.

A data platform is assembled from about seven primitives. Every cloud sells all seven under different names, and the names are the least interesting part of the comparison.

Six axes that actually separate analytical warehouses — architecture, storage/compute coupling, latency profile, concurrency model, cost-model shape and workload fit — and why a product name is the last thing to decide.

A serverless analytical engine: columnar storage you do not manage, compute allocated per query rather than provisioned, and exactly two physical knobs — partitioning and clustering — carrying all the layout weight.

Three separated layers — immutable columnar storage, independently sized compute clusters, and a services layer that holds all the metadata — and what that separation actually buys, which is isolation and elasticity rather than speed.

A columnar OLAP database built for logs, events and real-time aggregates: immutable sorted parts merged in the background, a sparse index over granules, and a sort order that decides almost everything.

An analytical database that runs inside your process. No network in the hot path, no cluster, no concurrency story — and that combination changes what a pipeline stage costs, not just how fast a query is.

Managed Kafka, Kinesis, Pub/Sub and Event Hubs are all realisations of the same primitive — a durable, replayable log. The axis on which they genuinely differ, and the one that changes your design, is ordering.

Eight questions that turn "which warehouse should we use" into a list of required capabilities, two or three candidate architectures, and the trade-off each one asks you to accept. The output is never a single product.

An agent system is a data platform with a model in the middle: six datasets, each with a grain, an owner, a freshness target, and its own quiet way of rotting.

Documents to ingest to clean to chunk to metadata to embed to index to retrieval. Nine stages, nine promises, and most retrieval failures happen in the first three.

Chunking is not preprocessing and not a hyperparameter. It is the grain declaration for the retrieval index, and a boundary in the wrong place is the same class of error as a wrong fact-table grain.

Turning a corpus into vectors is a batch job with a metered external call in the middle. Keyed sink, watermarked input, work queue derived by difference — or it will not finish.

A new embedding model makes every vector in the corpus stale. Recompute beside the old index and switch atomically — this is a data migration, and it obeys backfill rules exactly.

A vector is a row in a derived dataset. Source version, chunk strategy, embedding version, text hash and reindex status are the columns that make a corpus debuggable, rebuildable and governable.

Production traces, sampled, privacy-filtered and versioned into an evaluation dataset. The privacy filter is the step most often skipped, and the version is what makes a score comparable across runs.

Prompt, model, tool calls, latency, tokens, outcome and feedback — as a high-cardinality event table with a classification and a retention policy, not as logs in a bucket.

Raw events to transformations to features to two consumers. The characteristic failure is one logical column computed by two pipelines, and it is a data-engineering failure with a data-engineering fix.

The domain's closing question. Ten things you have to be able to answer about a figure before you are entitled to act on it — and what it means when you cannot answer one.

Finance and growth disagree about revenue. Both queries are correct. This is almost always a governance failure wearing the costume of a bug.

The report is low, no task failed, and the source database still has every record. Working from a shortfall back to the arrow that dropped it.

Revenue is up, nothing launched, and every check is green except uniqueness. Inflation is the failure people question least and notice last.

A complete, plausible, internally consistent number for a day that ended two days ago. The failure mode that looks most like health.

The domain's thesis, turned into a diagnosis. A green DAG proves the code ran; eight faults, six checks and the distinct fingerprint each one leaves are what prove anything else.

Sixteen decisions that were reasonable when they were made and expensive by the time anyone noticed. Each one gets the argument for it before the argument against.

The six failures that are organisational rather than technical. None of them is visible in a query plan, all of them are cheap in a small company, and each is what a large one means when it says the data cannot be trusted.

The seam is the write-ahead log. Almost every mechanism a source database uses to stay correct decides what a pipeline downstream of it is able to promise.

A pipeline does not choose its guarantees. It inherits them from the weakest hop, and most pipeline bugs are a distributed-systems property arriving where nobody expected it.

The backend produces transactions, events and logs as a side effect of serving users. We are its downstream consumer, and it usually does not know that.

A data platform is assembled almost entirely from four cloud primitives. Knowing which four, and what each actually charges you for, is most of platform engineering.

Transformation code deploys like software. The tables it already wrote do not, and that asymmetry is the whole lesson.

Observability & Performance owns why it is slow. This domain owns whether it is correct, complete and fresh. Different questions, different signals, different toolkits.

A pipeline is a machine for making copies. Every copy inherits the original's obligations and none of the mechanisms that were enforcing them.