ConnectionsGENERALSTAGE-SPECIFICILLUSTRATIVE

Problem Solving and Databases

Data → Access Patterns → Model → Index → Query. The database domain teaches models, indexes and query plans; this lesson is the step before — discovering that there is data, how it will be read and written, and only then which model and which index.

The moveWorked exampleNext questions

The situation, the reflex, and why it stalls

Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.

The question

The store has data and you need a schema. How does the loop get from plain-English requirements to a model you can hand to the database domain, and what decides the indexes before any query is slow?

The situation

You have written "Customer, Product, Order, OrderItem, Payment" on a whiteboard and stopped. You could draw a schema now, and it would look like every tutorial's schema. You are not sure it fits this store — prices that change, guest checkout, a report the founder wants — and you do not know how to find out before it is built.

The reflex

Draw the schema from the nouns. Five entities, obvious foreign keys, a diagram. Normalise it because that is what one does. The diagram is a real artefact and it looks like the data layer is designed.

Why it stalls

The schema encodes the nouns and none of the verbs. Nothing in the diagram says that an order's price must not change when the product's does, so the order item references the product's price and the first price change rewrites history (Snapshots vs References).

What the reflex produces — and fails to produce
  • The schema encodes the nouns and none of the verbs. Nothing in the diagram says that an order's price must not change when the product's does, so the order item references the product's price and the first price change rewrites history (Snapshots vs References).
  • Indexes are absent because no query exists yet, and when the queries arrive the indexes are added one slow page at a time. The access patterns were knowable on day one; they were never asked for.
  • The founder's report — revenue by day — was not in the nouns, so the schema cannot answer it without a scan of every order, and the analytics dashboard becomes a separate project to work around the schema.
  • Guest checkout requires a customer id that guests do not have, because the diagram drew the arrow from Order to Customer without asking whether every order has one.
ProblemUnderstandRequirementsConstraintsUnknownsDecompositionSmallest StepModelExperimentObserveDebugLearnIterate

The move

Precisely enough to apply it to a problem you have never seen — not a slogan.

  • Run the chain: Data → Access Patterns → Model → Index → Query. Data is the entities and the facts about them, from the requirements in plain English. Access patterns are the reads and writes the workflows need — which questions will be asked of the data, how often, and which writes happen together. The model is the shape that serves those patterns. Indexes are chosen from the patterns, not from slow queries. Queries are the last step, written against a model that was designed for them.
  • Get the access patterns from the workflows, not from imagination. Walk each core workflow — browse, add to cart, checkout, admin edits price, founder reads the report — and write the reads and writes it performs. The list is short and it is the specification for the model (Overwrite or Append?).
  • Let the patterns decide the contested modelling questions. Whether price is a snapshot or a reference is decided by "does the order history read need the price at order time?" — yes — not by normalisation doctrine. Whether the order needs a customer is decided by "does guest checkout create orders?" — yes — so the reference is optional (Data Modelling From Plain English).
  • Hand off at the model. Once the entities, the patterns and the snapshot decisions exist, the database domain's lessons on choosing a model, normalisation, indexes and query plans take over; they are far more useful with a pattern list than with a noun list.

The chain, and what each step hands the next

Five steps, of which the database domain owns the last three in depth. The first two are discovery, and they are what this domain contributes: finding out that data exists and how it will be used, before deciding its shape.

Data → Access Patterns → Model → Index → Query
  1. 1
    Data

    Entities and their facts, from the requirements in plain English; what must persist.

    fails by Nouns copied from a tutorial's schema.

  2. 2
    Access Patterns

    Per workflow: reads, writes, what happens together, how often, filtered by what.

    fails by Skipped; the model is drawn from the nouns.

  3. 3
    Model

    Entities shaped for the patterns: snapshot or reference, optional or required, one table or two.

    fails by Normalisation applied as doctrine; every reference a foreign key.

  4. 4
    Index

    One per frequent, filtered read; unique where the pattern demands it.

    fails by None until a page is slow; then one per slow page.

  5. 5
    Query

    Written last, against a model that already serves it; plan read when it surprises.

    fails by Written first, and the model bent to fit.

The chain is short on a store and takes an afternoon. Its output is a schema whose every decision has a pattern as its reason, which is what makes the schema changeable when the patterns do.

What the patterns leave unknown

Walking the workflows produces patterns and also exposes what nobody has decided. The board below is the store after the access-pattern step: what is now known, what is still assumed, and the unknowns each sharpened into a question the model needs answered.

The store, after the access patterns
known
  • Checkout writes order, order items and a stock decrement together, and must be atomic.
  • Order items need the price at order time; product price changes must not alter history.
  • The provider callback looks up an order by the provider's reference, so that reference must be unique and indexed.
assumed
  • ~One warehouse: stock is a single count on Product. Marked, because multiple warehouses turn it into a table.
  • ~A cart is not an order until checkout, and abandoned carts need not persist beyond the session.
unknown → question → experiment
  1. ? The report.

    becomes Is "revenue by day" by order date or by payment date, and does it include refunds?

    experiment Ask the founder with two example days that differ; put a paid-at timestamp on Payment if the answer is payment date.

  2. ? Guests.

    becomes Does a guest order need any customer record at all, or is an email on the order enough for confirmation and lookup?

    experiment Write the confirmation email and the "find my order" form on paper; see whether order number plus email covers both.

  3. ? Stock and concurrency.

    becomes When two checkouts decrement the last unit at once, does the database prevent the second, and how?

    experiment One row, two concurrent decrements, with and without a check constraint; observe the loser (Invariants Under Concurrency).

Each unknown changes the model — a column, a nullability, a constraint. Resolving them before the first migration is the whole payoff of the chain.

The model, as it would be written

The output of the chain, as data-definition language, with the decisions annotated by the pattern that made them. The point is not the SQL; it is that every non-obvious line has a reason that is not "that is how schemas look".

The store's model, with reasons
1create table product (
2 id bigint primary key,
3 name text not null,
4 price integer not null, -- current price; history lives on order_item
5 stock integer not null check (stock >= 0) -- "never oversell", enforced here
6);
7create index product_name_lower on product (lower(name)); -- name search, every keystroke
8create table "order" (
9 id bigint primary key,
10 customer_id bigint null, -- guest checkout: orders without a customer
11 email text not null, -- confirmation and lookup for guests
12 status text not null,
13 created_at timestamptz not null
14);
15
16create table order_item (
17 order_id bigint references "order",
18 product_id bigint references product,
19 quantity integer not null,
20 unit_price integer not null -- snapshot: order history must not change with price
21);
22create table payment (
23 id bigint primary key,
24 order_id bigint references "order",
25 provider_ref text not null unique, -- callback lookup; unique so a repeat cannot match twice
26 status text not null,
27 paid_at timestamptz null -- revenue by payment date, per the founder
28);
29create index payment_paid_at on payment (paid_at);

Notice which lines came from patterns rather than from the entity list: the check constraint, the lowercased name index, the nullable customer, the price snapshot, the unique provider reference, the paid-at index. None is visible from the nouns alone.

How to do it

Most important first.

  • List the entities from the requirements, then for each write the facts that must be true about it and the ones that change over time (Entities From Requirements).
  • For each core workflow, write its reads and writes as sentences: "checkout reads the cart items and current stock, writes an order, order items and a stock decrement, in one transaction".
  • Mark each read with how often it happens and what it filters by. The filters are the index candidates; the frequencies are the priorities (Should I Add an Index? in the database domain decides).
  • For every reference between entities, ask whether the referencing side needs the value as it was or as it is. "As it was" is a snapshot column; "as it is" is a foreign key.
  • Write the founder's report as a query in words before the schema is final. If the model cannot answer it without scanning everything, the model is not done.

Worked on a concrete problem

The move has to produce something. This is what it produced.

  • The store, run through the chain. Data: Product (name, price, stock), Order (status, created, customer optional), OrderItem (product, quantity, price at order time), Payment (order, provider reference, status). Access patterns: list products by name filter, every keystroke; read one product, every page view; checkout writes order + items + stock decrement atomically; provider callback finds an order by provider reference; founder reads revenue by payment date. Model: price snapshot on OrderItem; customer nullable on Order; Payment carries the provider reference. Indexes, from the patterns: products by lowercased name; orders by provider reference (unique); payments by paid-at date. Queries: written last, against a model that already has what they need.
  • The report, discovered as an access pattern. "Revenue by day" is asked of the founder — by order date or payment date? Payment date, because refunds and late confirmations move money after the order. That single answer puts a paid-at timestamp on Payment and an index on it, and the analytics dashboard is a query instead of a project (Case: An Analytics Dashboard).
  • The chat app: messages by conversation, newest first, paginated; unread count per member. The second pattern decides the model — a per-member last-read pointer, not a flag on every message — and the index — messages by (conversation, sent-at). The database domain's composite-index lesson explains why that column order; this lesson only says the pattern demanded it.

How you know it worked

What now exists that did not before, and what question you can now ask.

  • A written list of reads and writes per workflow exists before the schema diagram does.
  • Every snapshot-versus-reference decision has an access pattern as its reason.
  • Every index in the initial schema can be paired with a read pattern and its frequency.
  • The founder's report is answerable by the model without a full scan, and you knew that before the first row was inserted.

The questions you can now ask

The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.

Next questions
  • ?What are the entities, and which facts about each must be true and which change over time?
  • ?For each core workflow, what does it read, what does it write, and what happens together?
  • ?For each reference, does the referencing side need the value as it was or as it is?
  • ?Which reads are frequent enough and filtered enough to deserve an index on day one, and which can wait for evidence?

What can go wrong

How the move itself fails
  • Access patterns are enumerated for every conceivable query, and the model is designed for reports nobody asked for. The list is the core workflows plus the reports actually requested; everything else is a later index.
  • Indexes are added for every filter in the pattern list, including ones run once a month. Frequency is part of the pattern; an index is a write cost paid on every insert.
  • The chain is run once and frozen. New workflows bring new patterns, and a model designed for V1's reads will need revisiting when returns or multiple warehouses arrive (When Assumptions Change).
  • The handoff is skipped and the engineer re-learns normalisation from the schema's bugs. The database domain has a lesson for each decision; read it once the pattern list exists.
What the move costs
  • Writing the access patterns first delays the diagram, and a diagram is what the team expects to see when someone says the data layer is designed.
  • A model designed for known patterns is worse at unknown ones; a fully normalised model with no snapshots is more flexible and wrong about prices. The chain chooses the first and writes down why.
  • Indexes chosen from patterns can be wrong when the real frequencies differ from the guessed ones; they are cheap to drop, but only if someone measures.
Misreads
  • "So denormalise early." Snapshot where a pattern needs history; reference everywhere else. The chain produces one or two snapshots on a store, not a denormalised schema.
  • "Index every column that appears in a filter." Index the filters on frequent reads. The pattern list carries frequency for exactly this reason.
  • "This is database design." It is the step before: discovering what the database must serve. The design — model choice, normal forms, plan reading — is the database domain's, and it is linked, not restated.

Where this applies

Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

  • GENERALData → Access Patterns → Model → Index → Query holds for any persistent store; a document or key-value model changes what "model" means but not that the patterns decide it.
  • STAGE-SPECIFICGreenfield, the chain designs the schema. In an existing system, the model exists and the chain becomes: what new pattern does this feature add, and does the existing model serve it or need a migration (From Requirements to Tables and the migration lessons in the database domain).
  • ILLUSTRATIVEThe store's entities, the founder's report and the chat app's unread count are invented to show the chain producing a model; the frequencies are for the shape of the argument.

Where the depth lives

This domain asks the question and hands the answer off by name.

Further
  • The database domain's "requirements to schema" lesson continues from this one; arrive with the pattern list and the snapshot decisions, and the rest of the schema follows.