The Self-Service Data Platform

1. 1
   Search first

   Shows existing products matching the concept before anything is created.

   guarantees Nothing on its own — it makes the alternative visible, which is the only defence against duplicates that does not involve an approval (Data Discovery).

   fails by Being a separate tool nobody opens, so the first duplicate is created by someone who genuinely could not find the original.

2. 2
   Declare

   The engineer writes one file: source, model, owner rota, grain, tests, SLO, classification, retention, expiry.

   guarantees That everything the platform will need to govern this dataset exists before the dataset does (Data Products).

   fails by Accepting vacuous values — an owner that is a mailing list, a grain that says "various" — which passes validation and certifies nothing.

3. 3
   Validate

   Checks the declaration: does the source exist, is the SLO achievable, does the classification cover every column, do the tests reference real fields?

   guarantees Failures surface in seconds, at the moment the engineer is looking at the file, rather than at 03:00 next Tuesday (Data Tests).

   fails by Validating shape rather than feasibility, so a declaration promising an hourly SLO on a nightly source is accepted and breaks in production.

4. 4
   Generate

   Produces the ingestion, transformation wiring, tests, schedule, monitors, lineage capture and access grants.

   guarantees Every generated pipeline has the same failure semantics, the same retry behaviour and the same instrumentation (Idempotent Data Pipelines).

   fails by Allowing hand edits to generated artefacts, after which regeneration is unsafe and the platform can never change a default again (Drift: When the File and Reality Disagree).

5. 5
   Run and gate

   Executes on the shared orchestrator; tests and the conformance gate run before anything is published.

   guarantees Consumers see a version that passed its declared assertions, published atomically (Atomic Publish).

   fails by A gate that warns instead of blocking, which is indistinguishable from no gate within a month (Alert Fatigue: The Page Nobody Reads).

6. 6
   Register and monitor

   Publishes to the catalog with lineage, freshness and volume monitors, alert routing to the declared rota, and cost attribution.

   guarantees The dataset is findable, watched, attributable and has a name attached to its pager (The Data Catalog, Freshness Monitoring).

   fails by Routing alerts to the platform team by default, which absorbs ownership the platform cannot discharge (Data Ownership).

7. 7
   Expire or renew

   At the declared date, asks the owner to renew; retires the dataset if nobody does, with a restore window.

   guarantees The catalog has an equilibrium rather than a growth rate. This is the stage that prevents the four-hundred-dataset outcome (Storage Lifecycle).

   fails by Retiring something with a quiet but important consumer and having no restore path, after which the whole mechanism is switched off permanently.

Count the stages that require a person outside the requesting team: none. That is the definition being operationalised — not "there is a documented process", but "no queue exists between intent and production".

The platform publishes a starter project and a wiki page. Teams copy it, edit the pipeline code directly, request a warehouse grant, ask for a scheduler slot, and open a ticket for the source credential. Governance is a review at the end, when it happens.

Teams write one declaration. The platform validates it, generates every artefact, derives grants and masking from the declared classification, registers lineage, routes alerts to the declared rota, attributes cost, and sets an expiry. Generated artefacts are never edited by hand; the declaration is changed and everything is regenerated.

A template is copied once and then diverges, so the platform loses the ability to know what exists, to change a default, or to enforce anything after creation day. A declaration stays the source of truth, which is what makes validation, regeneration, enumeration and retirement possible at all — and those four capabilities are the difference between a catalog that can be governed and one that can only be surveyed (Drift: When the File and Reality Disagree).

1# products/orders/orders_daily.yaml
2# The whole interface a domain engineer touches. Everything below is required;
3# the platform refuses to generate a pipeline from an incomplete declaration.
4 
5product: orders_daily
6owner:
7  team: orders
8  rota: orders-primary          # must resolve to a pager, not a mailing list
9grain: >
10  One row per placed order per calendar day of placement, keyed on
11  (order_id, order_date). Cancellations appear as a later row, never
12  as a mutation of an earlier one.
13 
14source:
15  kind: cdc                     # platform owns the connector and checkpointing
16  system: orders_postgres
17  tables: [orders, order_lines]
18 
19model:
20  sql: models/orders_daily.sql  # domain logic, in the domain's own repository
21 
22slo:
23  freshness: orders placed before midnight are complete by 06:00 local
24  completeness: reconciled against the source for every closed day
25 
26tests:
27  - unique: [order_id, order_date]
28  - not_null: [order_id, order_total_minor, order_date]
29  - accepted_values: {column: status, values: [placed, shipped, cancelled]}
30  - volume_vs_history: {column: order_id, window: 28d}
31 
32classification:
33  contains_personal_data: true
34  fields:
35    customer_email: contact
36    shipping_postcode: location
37retention: 24 months
38access:
39  read: [analytics, finance]
40 
41expires: 2027-03-01             # renewed by the owner, or retired automatically
42 
43# The platform generates from this: the ingestion job, the transformation
44# wiring, the schedule with retries and backfill ranges, the test suite in the
45# publish path, freshness and volume monitors routed to orders-primary, the
46# catalog entry with lineage, the grants and masking implied by the
47# classification, the retention rule, and the cost attribution tag.

The interesting property is what a missing field does. No rota means no pipeline, so ownership cannot be deferred to later — the moment when someone is motivated to fill it in is the moment they want the dataset. An expiry date is required for the same reason: retirement is negotiated once, at creation, rather than four hundred times, under pressure, two years later.

Five checks, all of them queries over metadata the platform already collects, and all of them blind to the same thing: whether the dataset means what its name says. That is the residue of judgement a self-service platform concentrates rather than removes.