Data Engineering and Security
A pipeline is a machine for making copies. Every copy inherits the original's obligations and none of the mechanisms that were enforcing them.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What changes about access, classification, retention and deletion once data leaves the system that was protecting it?
Everyone who will be asked to prove something later: the engineer answering a deletion request, the auditor asking who read a table last quarter, the team that must show a dataset never left a jurisdiction. All three need answers from records that only exist if somebody designed them in advance (Audit Trails).
The unit that matters here is one column in one copy of one dataset. Classification is a column-level property — one email field makes an otherwise innocuous table a personal-data asset — and copies are the thing that multiply, so the security question is always "this column, in how many places" (Column-Level Lineage).
Rely on the source system's access controls and on the warehouse's roles. The application enforced who could see a customer record; the warehouse has role-based grants; between the two, coverage looks complete.
The raw landing zone holds the full source payload including fields nobody selected, because landing raw is correct engineering practice. Those fields include a national identifier the application only ever displayed to two internal roles (The Raw Landing Zone, Data Minimization).
- The raw landing zone holds the full source payload including fields nobody selected, because landing raw is correct engineering practice. Those fields include a national identifier the application only ever displayed to two internal roles (The Raw Landing Zone, Data Minimization).
- A deletion request removes the customer row from the source. It remains in the lake, in the warehouse, in three snapshots, in the event log until retention passes, in a training dataset and in an analyst's extract (Deletion Requests).
- A pipeline's service identity has broad read access because that is what got the first job working. Nobody can answer "which jobs can read the PII bucket", and narrowing it now risks breaking jobs nobody owns (Data Access Control, Least Privilege).
- An error path logs the failing record to make debugging possible. The record is a payment, the log is retained centrally, and personal data is now in a system with an entirely different access model (Secrets in Logs, Security-Safe Logging).
- A dataset is masked in the curated layer and unmasked in staging, and staging is readable by everyone because it was "just an intermediate step" (Data Masking, Tokenisation & Encryption, Raw, Staging, Curated: Layers by Purpose).
- An embedding pipeline turns support tickets into vectors in a store with no row-level access control, and the vectors are close enough to the source text to be a disclosure (Embedding Pipelines, Vector Data Engineering).
What is actually happening
- Copying is the whole problem. A data platform exists to move data out of the system that was protecting it, and every hop creates an artefact — a file, a topic, a table, a snapshot, an extract — with its own access model. Security properties do not travel with the bytes; they have to be re-established at every stop (Data Governance).
- Classification is what makes any of the rest mechanisable. Until each column is labelled, "protect the sensitive data" is not an instruction a pipeline can follow. Classification propagates through lineage: a model built from a classified column inherits the classification unless a transformation demonstrably removes it (Data Classification, Data Lineage).
- Minimisation is the strongest control available and the cheapest, because it removes the obligation rather than managing it. A field never landed needs no policy, no masking, no retention rule and no deletion path (Data Minimization).
- Access control in analytics is coarse where application authorization is fine. An application authorizes per object and per user; a warehouse grants per table, per column and per row policy; an object store grants per bucket and prefix. Reproducing per-record application logic in analytics does not work, which is why the answer is usually a masked or filtered view rather than a translated policy (Row and Column Security, Object-Level Authorization).
- Deletion is the obligation that pipelines are structurally worst at. Immutable files, append-only logs, snapshot history and columnar layouts are all designed to make rewriting rows expensive. That is why deletion strategy — partition alignment, table-format deletes, or crypto-shredding by key scope — is a design decision taken before the first row lands (Deletion Requests, Key Management and Encryption at Rest).
- Retention is a security control and a cost control at once, and it is the only one where those two interests point the same way. Shorter retention reduces exposure and spend, and reduces the window in which a mistake is fixable — which is the genuine tension (Data Retention, Retention and Replay).
- Aggregation is not anonymisation. A metric grouped by a small segment, joined against another metric, re-identifies individuals; and a pseudonymous key that is stable across datasets is a join key, which is exactly what re-identification needs (PII in Pipelines).
Where the boundary sits
Security Engineering asks how a system is attacked and how it is defended: threat models, authentication, authorization, cryptography, exploitation. This domain asks a narrower question with a wider surface — what happens to those defences when the data is copied out of the system that implemented them, six times, into artefacts with six different access models.
The framing that makes this tractable: the attacker's view is theirs; the copy count is ours. We are not going to out-think a security engineer about a vulnerability class. What we know that they cannot is where every copy of a column lives, which pipeline made it, and what would have to happen for it to be deleted.
That is also why lineage and classification are the two mechanisms that do the most work on this boundary. Neither is a security control. Both are what makes a security control applicable to a platform whose defining behaviour is duplication.
| We teach | Depth lives in | The mechanism that crosses |
|---|---|---|
| Labelling what is sensitive | Security: Data Classification, Threat Modeling: The Process | Classification is per column and propagates along lineage. Until it exists, no pipeline rule can be written at all (Column-Level Lineage). |
| Deciding who can read a dataset | Security: Authorization Models, Role-Based Access Control, ABAC and Policy-Based Authorization, Least Privilege | Application authorization is per object and per user; analytics is per table, per column and per row policy. The translation is a masked view, not a ported rule (Row and Column Security). |
| Not landing what you do not need | Security: Attack Surface, Defense in Depth | Minimisation removes the obligation instead of managing it, and it is the only control on this list with no ongoing cost (Data Minimization). |
| Encrypting datasets and scoping keys | Security: Encryption at Rest vs in Transit, Symmetric Encryption, Secrets Management | Encryption is easy; key *scope* is the data decision, because per-subject or per-tenant keys make deletion a key operation (Deletion Requests). |
| Deleting a subject from a platform | Security: Data Classification, regulatory obligation | Immutable files, append-only logs and snapshot history are all designed to resist row deletion. The deletion path is a storage-layout decision made in advance. |
| Proving who read what | Security: Audit Logs for Privileged Actions, Security-Safe Logging, Detection Engineering | Access logs on classified datasets, retained longer than the data, are the only way to answer the question that is always asked after the fact (Audit Trails). |
| Keeping data inside a boundary | Security: Network Segmentation, Egress Security, Trust Boundaries | Region, account and network boundaries are where a jurisdictional constraint becomes enforceable rather than documented (Data Governance). |
| Corpora and embeddings for retrieval | Security: RAG and Agent Memory Security, Prompt Injection | An embedding is a derived copy that inherits the source's classification, and a vector store often has coarser access control than the table it came from (Embedding Pipelines). |
One column, counted in copies
The single most useful exercise on this boundary is to take one sensitive column — email will do — and enumerate every artefact in the platform that holds it or something derived from it. Nearly every team that does this finds copies they did not know about, and the copies are almost always in the layers nobody classified.
The chain below is that enumeration for a typical platform. The couldCorrupt column is used here in its security sense: what this node can leak, retain past its obligation, or fail to delete. Read it as a checklist for a deletion request.
Notice that the highest-risk node is the one built for the best engineering reason. Raw exists so transformations are recoverable, which means it holds the complete source payload — including the fields the application never exposed — and it is retained the longest of anything in the platform (The Raw Landing Zone, Data Minimization).
- Source table column
holds The authoritative value, protected by the application's per-user authorization.
could corrupt Nothing new — but its deletion here is the event everything downstream is supposed to react to, and nothing downstream reacts automatically (Deletion Requests).
↑ reads from - Change event in the log
holds Every historical value of the column, including ones since overwritten.
could corrupt Retains superseded values for the whole retention window and is replayable by any consumer group; a deletion in the source is just another event, not an erasure (Retention and Replay).
↑ reads from - Raw landing objects
holds The complete source payload, including fields no model uses.
could corrupt Holds columns nobody classified, retained longest, readable by whichever identities have the bucket prefix; deleting one subject means rewriting immutable objects (Object Storage as Data Infrastructure).
↑ reads from - Staging model
holds Typed, deduplicated rows — often still unmasked, because masking is applied later.
could corrupt The classic gap: readable by everyone with warehouse access because it is "intermediate", while the curated layer above it is carefully masked (Raw, Staging, Curated: Layers by Purpose).
↑ reads from - Curated dimension
holds The modelled, masked or tokenised customer attributes.
could corrupt A pseudonymous key that is stable across datasets becomes a join key for re-identification; SCD2 history preserves values the subject asked to have removed (SCD Type 2 in Practice).
↑ reads from - Snapshots and object versions
holds Point-in-time copies of every table above, kept for rollback.
could corrupt Retains deleted values after the live table no longer has them, and is routinely excluded from retention policies because it is thought of as infrastructure (Storage Lifecycle).
↑ reads from - Embedding / feature store
holds Vectors and features derived from the same text or attributes.
could corrupt Inherits the classification and usually has coarser access control; embeddings can be close enough to the source to constitute disclosure (Embedding Pipelines, Feature Pipelines).
↑ reads from - BI extracts and notebooks
holds Whatever a consumer downloaded, wherever they put it.
could corrupt Leaves the platform entirely — no policy, no retention, no audit, and invisible to lineage. This is where most real exposure is (Data Discovery).
A deletion request has to reach every node on this list. Count the nodes in your own platform before promising a timeline.
Applying the policy at read time
The instinct when a column is sensitive is to build a masked copy in the pipeline. It is a natural data-engineering move and it makes the problem worse: there are now two physical tables, the unmasked one is upstream, and it is readable by everyone who can read the layer it lives in.
The better shape is one physical table with the policy applied in the read path — a view, a row-access policy, or a column mask enforced by the engine. There is exactly one copy to protect, one place to audit, and a change to the policy takes effect without recomputing anything.
The SQL below is deliberately plain and portable: the grant table *is* the policy, and the outer join is the enforcement. Note the failure direction — a principal with no matching grant row gets a null rather than an error, which fails closed for exposure and open for availability. That choice belongs in a review, not in a default (Fail Open vs Fail Closed).
A subject asks for erasure. Which mechanism removes their rows from every copy?
when The data is partitioned by a key that aligns with the deletion predicate — most often a tenant rather than an individual. Erasure is a metadata operation (Partitioning).
cost Only works when the layout was chosen for it, and partitioning by subject is almost always terrible for query performance and file count (Partition Cardinality).
when Tables are managed by a format that records deletes in metadata and applies them during compaction (Open Table Formats).
cost Cheap at request time and paid later in compaction; the values persist in old snapshots until those expire, so snapshot expiry becomes part of the compliance path (File Compaction).
when Bare files with no table format, and the volume of requests is low enough to absorb the rewrite.
cost Rewrites every object containing a matching row, invalidates readers mid-operation without an atomic publish, and scales badly with request volume (Atomic Publish).
when Data was encrypted per subject or per tenant from the start, so destroying the key renders every copy unreadable at once, everywhere (Key Management and Encryption at Rest).
cost Must be designed before the first row lands and cannot be retrofitted; key management becomes a critical availability dependency, and a lost key is an unrecoverable dataset.
when The field has no analytical use that justifies the obligation. Drop it at ingestion (Data Minimization).
cost Irreversible: the question you cannot ask next year is invisible today, and this is the one decision in this domain where "keep everything" is the wrong instinct.
1-- One copy of the data. The grants table is the policy; the join enforces it.2create view analytics.dim_customer_v as3select4 c.customer_id,5 c.customer_key_hash, -- stable pseudonym, per-dataset salt6 case when g.principal is not null7 then c.email end as email,8 case when g.principal is not null9 then c.postcode10 else substr(c.postcode, 1, 3) end as postcode_area,11 c.country_code,12 c.signup_date13from analytics.dim_customer c14left join governance.pii_grants g15 on g.principal = current_user16 and g.dataset = 'dim_customer'17 and g.valid_to > current_timestamp;18 19-- Consumers are granted on the view only. A direct grant on20-- analytics.dim_customer silently defeats all of the above, which is why21-- base-table grants belong in the same review as the policy itself.The thing to notice is what is absent: there is no second, masked table. A pipeline stage that wrote one would leave analytics.dim_customer unmasked and readable by anyone with access to that schema — and the masked copy would then need its own retention and deletion path. Per-dataset salting of the pseudonym is what stops the hash becoming a universal join key across datasets (Data Masking, Tokenisation & Encryption).
How to build it
Most important first.
- Classify at ingestion, at column level, and carry the classification as metadata through every derived model. A label applied at the warehouse edge has already missed the raw layer (Data Classification, Metadata: Technical, Operational and Business).
- Minimise on the way in wherever the field genuinely has no analytical use, and be honest that this trades away future questions you cannot ask (Data Minimization).
- Give every pipeline its own workload identity with the narrowest grant that lets it run, and no long-lived static credentials (Roles vs Static Keys, Least Privilege in Infrastructure).
- Expose sensitive datasets through masked or row-filtered views, so there is one physical copy and the policy is applied at read time rather than by a pipeline step that creates a second, unmasked table (Row and Column Security, Data Masking, Tokenisation & Encryption).
- Design the deletion path before the first row lands: which partition a subject's rows fall in, whether the table format supports row-level deletes, and whether key scope allows crypto-shredding (Deletion Requests).
- Log access to classified datasets and retain those logs longer than the data itself, because the question "who read this" is asked after the data is gone (Audit Logs for Privileged Actions, Audit Trails).
- Keep secrets out of pipeline code and out of logs, and treat a failing-record log line as a data export (Secrets Management, What You Just Wrote Into a Log Half the Company Can Read).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- The source system's authorization guarantees nothing at all about the copy. It was enforced by code that the pipeline does not run (Trust Boundaries).
- Encryption at rest guarantees protection against physical media access. It guarantees nothing against a credential that is allowed to read the data, which is the actual threat model for an analytical platform (Encryption at Rest vs in Transit).
- A masked view guarantees that a reader of that view does not see the raw value. It guarantees nothing about the underlying table, any other view over it, or the raw layer it came from — which is why the physical copy is the thing to count (Data Masking, Tokenisation & Encryption).
- A retention policy guarantees deletion of what it covers. Snapshots, object versions, replicas, backups and downstream extracts are separate artefacts and need their own policy (Storage Lifecycle, Backups Are Sensitive Data Copies).
- A deletion of a source row guarantees nothing downstream. There is no automatic propagation of erasure through a pipeline; every hop needs an explicit mechanism.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that belongs here is a classified-column inventory: enumerate every column across every layer that lineage says derives from a classified source column, and diff it against the set of tables where a policy is actually applied.
- It catches the derived table nobody labelled, the staging copy with no policy, and the extract that reproduced a masked column unmasked. It misses anything lineage did not capture — a hand-run script, a notebook, a spreadsheet export — which is where the real exposure usually is (Data Discovery).
- The complementary check is a scan for high-signal patterns in columns that are *not* classified, which catches personal data arriving in a free-text field and misses anything that does not match a pattern.
- Deletion has a latency of its own, and it is usually the slowest operation in the platform: a request received today may not be reflected in every copy until the next compaction, the next snapshot expiry, or the next full rebuild. That interval is a commitment and should be stated (Deletion Requests).
- Access-policy changes propagate at different speeds by layer — a warehouse grant is immediate, an object-store policy is near-immediate, a materialised masked copy is not updated until it is rebuilt.
- Retention windows are the one freshness property that runs backwards: they decide how far into the past a consumer may look, and shortening one is a breaking change for anybody who was looking further (Data Retention).
- A source adding a field adds it to the raw payload whether or not anyone asked, so classification has to run on arrival rather than on a schema you approved (CDC and Schema Drift).
- A field can become sensitive without changing: a free-text
notescolumn that users start pasting identifiers into is now personal data with the same type and name it always had (Semantic Changes). - Regulatory scope changes are the one evolution here that no technical signal announces. The mitigation is that classification and lineage make the impact assessment mechanical rather than archaeological (Impact Analysis).
- Recovery from an over-broad grant is not only revoking it — it is determining from access logs what was read while it was open, which requires those logs to have existed (Audit Logs for Privileged Actions).
- Recovery from leaked data in a raw layer means rewriting or dropping the affected objects and every derived artefact, including snapshots and versions, which is exactly the operation the storage layer is designed to make expensive.
- Crypto-shredding makes some of this tractable: destroy the key for a subject or tenant and every copy encrypted under it becomes unreadable at once, wherever it is. It only works if key scope was chosen at design time (Key Management and Encryption at Rest).
What can go wrong
- Raw retaining fields the application never exposed, indefinitely, because raw retention was designed for reprocessing rather than for governance.
- A deletion request satisfied in the warehouse and not in the lake, the log, the snapshots or the training set.
- One broad pipeline identity that makes access unauditable and un-narrowable.
- Personal data in error logs, in a system with a different access model and a different retention policy.
- The mitigation failing: a masking step implemented as a pipeline stage that writes a masked copy, leaving the unmasked intermediate readable by anyone with staging access (Raw, Staging, Curated: Layers by Purpose).
- A pseudonymous identifier reused across datasets, which turns two separately-safe tables into a join that re-identifies (PII in Pipelines).
- "The warehouse has roles, so access is controlled." Warehouse roles govern the warehouse. The lake, the topic, the snapshots, the notebooks and the extracts are governed by other systems or by nothing (Data Access Control).
- "We deleted the row." You deleted one copy. Deletion in a pipeline is a propagation problem, and the propagation does not exist unless it was designed (Deletion Requests).
- "It is anonymised, it is aggregated." Aggregation over small groups, joined against other aggregates, re-identifies. A stable pseudonym is a join key, not an anonymisation.
- "Encryption at rest protects the data." It protects it from someone taking the disk. It does nothing against a credential with read access, which is the realistic threat to an analytical platform (Encryption at Rest vs in Transit).
- "Raw is internal, so it does not need controls." Raw is the most sensitive layer in the platform: it is the complete source payload, it is retained the longest, and it is the one nobody has reviewed the columns of (The Raw Landing Zone).
- Classification, minimisation, retention, access and deletion are the five obligations, and each one attaches per column and per copy rather than per system (Data Governance).
- Retention policy must cover every artefact separately: table, snapshot, object version, topic, backup, and any extract that left the platform (Data Retention).
- Key scope decided at design time is what makes crypto-shredding available as a deletion mechanism, and it cannot be retrofitted onto data already encrypted under a shared key (Key Management and Encryption at Rest).
Operating it
- A classified-column inventory across every layer, produced from lineage rather than maintained by hand (Column-Level Lineage).
- Access logs on classified datasets — who read, which columns, how much — retained longer than the data (Audit Logs for Privileged Actions).
- Grant drift: the diff between intended and actual access per classified dataset, evaluated on a schedule rather than at review time (Data Access Control).
- Deletion-request throughput and age: how many are open, and how long the oldest has been waiting for the next compaction (Deletion Requests).
- Egress signals: which identities exported which datasets to where, since the copy that leaves the platform is the one no policy reaches (Egress Security).
- At 10x datasets, manual classification stops happening. The platform needs classification applied at ingestion by default, with the unclassified state treated as restricted rather than as public (Fail Open vs Fail Closed).
- At 100x, or across many teams, per-dataset access requests become the bottleneck and the design shifts to attribute-based policies over classified metadata rather than per-table grants (Attribute-Based Access Control, ABAC and Policy-Based Authorization).
- Consumer count scales the exposure surface directly: every additional reader is another identity, another possible extract, and another copy the platform does not control (Data Discovery).
- Retained bytes are simultaneously the storage driver and the exposure driver, which is the rare case where the security argument and the cost argument agree (Storage Lifecycle).
- Row-level deletion in columnar, immutable storage costs a rewrite of every file containing a matching row, which is why partition alignment with the deletion predicate is a cost decision as much as a compliance one (File Compaction).
- Masked views cost query-time computation on every read instead of a one-off transformation cost, which is usually the right trade because it removes a physical copy (Data Masking, Tokenisation & Encryption).
- Audit logging costs retained bytes proportional to access volume, and is one of the few places where retaining more is the correct default (The Log Bill and What It Is Buying).
- Minimisation buys the strongest possible protection and costs the questions you cannot answer later. It is genuinely irreversible, and it is the one place in this domain where "keep everything" is bad advice (Keeping Raw History: The Recovery Position and the Liability).
- Fine-grained access buys auditability and costs friction on every new pipeline, which is why unpaved policy paths get routed around with a shared account (The Self-Service Data Platform).
- Short retention reduces exposure and reduces the window in which any mistake is repairable by replay. That tension is real and is decided per dataset, not per platform (Retention and Replay).
- Masked views remove a physical copy and put policy in the read path, at the cost that every consumer must be routed through the view — and a single direct grant on the base table quietly undoes it.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThat copies inherit obligations and lose enforcement, that classification is a column-level property, and that deletion does not propagate on its own are true of every platform in every jurisdiction.
- ORG-SPECIFICWhich data category triggers which obligation, how long a deletion request may take, and whether pseudonymised data still counts as personal are legal determinations that vary by jurisdiction and by sector. Engineering can only make the mechanism available; it cannot decide the threshold, and a platform designed against one regime does not automatically satisfy another.
- FORMAT-SPECIFICDeleting one subject's rows from bare Parquet means rewriting every file that contains one; a table format with row-level deletes records the deletion in metadata and applies it at compaction, so the same request is cheap immediately and expensive later. The obligation is identical, the mechanism and its cost are not (Open Table Formats).
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns how policy itself is delivered — access rules as code, reviewed and applied through a pipeline rather than clicked into a console — which is what makes grant drift detectable instead of discovered.