Property-Based Testing
When a behaviour can be stated as something true of every input, you can test the statement instead of a handful of examples — and being unable to state one is itself a finding about the design.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
Can this behaviour be expressed as something that must hold for all inputs, and what does it mean if it cannot?
A discount engine has 40 example tests. A customer reports a negative total. Nobody can say which combination of coupon, tax and rounding produced it, and no example covers it.
Write more examples. Every bug found in production becomes a new test case, and over time the examples cover the space.
Examples cover the space you thought of, and the bug is by definition in the part you did not. Forty examples over four interacting rules is a rounding error on the combination space.
- Examples cover the space you thought of, and the bug is by definition in the part you did not. Forty examples over four interacting rules is a rounding error on the combination space.
- Each example encodes a specific expected output, so when the rule changes, all forty need recomputing by hand — the maintenance cost grows with coverage (Change Amplification).
- Examples do not state the rule. Reading forty of them, a new engineer can infer that totals seem positive; nobody has said it must be.
- The examples that get added after incidents are the narrowest possible: exactly the reported input, which is the one case now guaranteed never to recur.
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- Property tests are non-deterministic by nature; a CI that must be perfectly reproducible needs seeds recorded and replayable.
- They are slower than examples — hundreds of runs per property — so the whole suite cannot be properties.
- The team has to be able to read a failure. A minimal counterexample is readable; a random 200-element input is not.
- A total is never negative. That is the property, and it was always true — it was just never written down anywhere executable (Invariants).
- A property must be genuinely universal. A property with an unstated exception is a false statement that will eventually be found by the generator, which is embarrassing and also useful.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The property owns stating a truth about the system in the domain's own words — "a total is never negative", "applying discounts in any order gives the same result".
- The generator owns producing inputs that are valid, which forces you to say what "valid" means — often the most valuable output of the whole exercise (Making Illegal States Unrepresentable).
- Example tests keep their job: pinning specific known cases, especially the ones from incidents, where the value is the specificity.
- Properties fit where behaviour is a function of inputs — the pure core. They fit badly where behaviour is a sequence of effects, unless you model the effects as a state machine (Functional Core, Imperative Shell).
- The generator boundary is where "what is a valid input" gets defined, and pushing that definition into the type system means the generator gets simpler and the production code gets safer at the same time (Value Objects).
- A property is a contract at exactly the same boundary a unit test uses — it just quantifies over inputs instead of naming one (What a Unit Is).
From forty examples to one statement
The discount engine bug is a good specimen: nobody had written down that a total cannot be negative, because it was obvious. Obvious rules are exactly the ones that go unstated, and unstated rules are the ones the fifth interacting feature breaks.
The property below is three lines and covers more of the input space in one run than the forty examples do in total. Note that it does not compute an expected total — that would just be the implementation twice.
1// invariance — the rule nobody wrote down2property('a total is never negative', gen.order(), (o) => {3 expect(price(o).total).toBeGreaterThanOrEqual(0)4})5 6// commutativity — order of discounts must not matter7property('discount order is irrelevant', gen.order(), gen.perm(), (o, p) => {8 expect(price(o).total).toBe(price(withDiscountsIn(o, p)).total)9})10 11// metamorphic — a relation between two runs, no expected value needed12property('adding a line never lowers the total', gen.order(), gen.line(),13 (o, l) => {14 expect(price(add(o, l)).total).toBeGreaterThanOrEqual(price(o).total)15 })16 17// round trip — the classic, and it finds encoding bugs immediately18property('an invoice survives serialisation', gen.invoice(), (i) => {19 expect(parse(serialise(i))).toEqual(i)20})None of these states an expected output, which is what keeps them from being the implementation restated. The metamorphic shape is the most transferable idea here: you can assert a *relationship between two runs* without being able to say what either run should return, which is what makes properties usable on messy business rules (Invariants).
Properties over a lifecycle
The technique extends past pure functions. Generate a random sequence of valid commands against a state machine, run it against the real implementation and against a tiny model of the intended behaviour, and assert they agree — and, more cheaply, assert that no forbidden state was ever reached.
This is where property testing and state design meet: the forbidden transitions below are the properties. Writing them down is a design act that pays off whether or not you ever run the generator (Invalid Transitions).
| From | On | To | Guard | Effect |
|---|---|---|---|---|
| TRIALING | trialEnded | ACTIVE | a payment method exists | first charge is attempted |
| TRIALING | trialEnded | CANCELLED | no payment method | — |
| ACTIVE | pause | PAUSED | pauses used this year < 2 | renewal date shifts by the pause length |
| PAUSED | resume | ACTIVE | — | — |
| ACTIVE | chargeFailed | PAST_DUE | — | dunning schedule starts |
| PAST_DUE | chargeSucceeded | ACTIVE | — | — |
| PAST_DUE | dunningExhausted | CANCELLED | — | — |
| ACTIVE | cancel | CANCELLED | — | — |
- CANCELLED → ACTIVE — Resurrection would reuse an id whose billing history is closed, so charges would attach to a period that has already been invoiced and reconciled. A resubscribe must mint a new subscription.
- PAUSED → PAST_DUE — A paused subscription is not billed, so a failed charge here means something billed it anyway — the property catches a bug in the biller, not in the state machine (Idempotency by Design).
- TRIALING → PAUSED — Pausing a trial has no defined meaning: the trial is time-boxed, so a pause either extends free access indefinitely or does nothing. Undefined behaviour that would be discovered by a customer.
- PAST_DUE → PAUSED — It would let a customer escape dunning by pausing, freezing the debt while retaining access. A revenue bug that no example test would think to write.
The generator emits random valid command sequences; the property is that no run ever produces a state pair from the forbidden list, and that the implementation's state matches a ten-line model at every step. Three of the four forbidden transitions above are the kind of thing that is obvious once written and invisible until then (State Machines).
When the property will not come
Sometimes you sit down to write a property and cannot. That is not a failure of the technique, and the temptation — write a weak property that restates the code — should be resisted, because it produces a test that passes forever and means nothing.
The interesting cases are the ones where the difficulty is a design signal rather than a domain fact.
looks like Every attempt at a universal statement needs an exception clause, or the only "property" anyone can write recomputes the expected value using the same logic as the implementation.
suggests Either the behaviour is genuinely a table of decisions with no underlying rule — which is common and fine — or the code has conflated several rules that each have a clean property, and the conflation is why no statement covers the whole thing (Separation of Concerns).
fix Try to split first: pricing usually decomposes into a lookup (a table, tested by example) and an arithmetic combination (rules, tested by properties). If the split works, both halves get better tests than the fused version could. If it does not, use examples and write the invariant down as an assertion in the production code instead (Enforcing Invariants).
How to build it
Most important first.
- Look for the classic shapes first: round trip (parse after serialise is identity), invariance (the total is never negative), commutativity or idempotence (order does not matter; applying twice equals once), oracle (the new fast implementation agrees with the old slow one), metamorphic (adding an item never decreases the total).
- State the property in domain language before writing any generator. If it cannot be stated in one sentence, that is the finding — the behaviour may not be well defined (Requirements Before Design).
- Constrain the generator to valid inputs, and notice how hard that is. A generator that needs twelve guards is telling you the type permits states the domain does not (Explicit State).
- Rely on shrinking. The value of a property failure is the *minimal* counterexample: "quantity 1, discount 100%, tax 20%" is a bug report; a random order with 47 lines is noise.
- Record the failing seed and add it as a permanent example test. The property found it once; the example makes sure it stays found (Characterization Tests).
- Use properties alongside examples, not instead. Properties state the rule; examples pin the cases people argued about.
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Adding a fifth discount type: with examples, recompute forty expected totals by hand and hope the interactions are covered. With properties, the existing properties immediately apply to the new type and either hold or produce a minimal counterexample — the cost of the next rule is roughly constant rather than proportional to the example count.
- Changing the rounding rule: examples all break at once and each needs a new hand-computed number. Properties about non-negativity, monotonicity and order-independence keep holding, and only the properties that actually mention rounding need attention.
- The permanent cost: a slower suite, a seed-management convention, and one more thing a new engineer has to learn to read.
- They are slower and non-deterministic, and a suite that occasionally fails on an unlucky seed erodes trust unless seeds are recorded and replayed.
- They demand that you can state the rule, and for genuinely messy domains — tax law, shipping surcharges — there may be no universal statement, only a table. Forcing a property there produces something false.
- Writing good generators is a real skill, and a bad generator is worse than no property: it costs runtime and gives confidence it has not earned.
What can go wrong
- The property is a restatement of the implementation — computing the expected value the same way the code does — so it passes by construction and tests nothing. This is the most common way property tests fail to be useful.
- The generator is too narrow, so the interesting region is never sampled and the suite gives false confidence at a much higher runtime cost than examples.
- The property is not actually universal, fails on a legitimate edge case, and gets weakened with guards until it says almost nothing.
- Shrinking is poor and the counterexample is unreadable, so failures get ignored or reruns are used until it passes — which converts the tool into flakiness.
- The suite gains a dependency on a generator library and on its shrinking quality, which varies enormously between ecosystems.
- Generators depend on the domain types. Good types make generators trivial; primitive-obsessed code makes them a project in themselves, which is a design signal (Primitive Obsession).
- CI depends on being able to replay a seed, or an intermittent failure becomes unactionable (A Deterministic Core).
- "Property tests replace example tests." They answer different questions. Properties say what must always hold; examples pin the specific cases a human argued about, and those arguments are worth preserving.
- "Just generate random inputs." Random input without a stated property is fuzzing — valuable for finding crashes, silent about correctness.
- "If I cannot find a property, the tool does not apply." Sometimes. But often it means the behaviour is not well defined, and that is worth knowing before it becomes a support ticket (Requirements Before Design).
- "Properties are only for parsers and data structures." Round trips and invariance show up everywhere: totals, permissions, state transitions, retries, id generation (Idempotency by Design).
Testing it, and how it ages
- A property per invariant, at the same boundary you would have written a unit test (What a Unit Is).
- An oracle property whenever you optimise: the new implementation must agree with the old one on every generated input, which makes performance work far less frightening (Premature Optimization, Reclaimed).
- A stateful property for lifecycles: generate random valid command sequences and assert that no forbidden state is reachable (State Machines).
- Every shrunk counterexample becomes a permanent example test, with a comment naming the incident.
- Properties age far better than examples, because they express the rule rather than a computed consequence of it. A five-year-old property is usually still true; a five-year-old expected total usually is not.
- The generator becomes a de facto specification of valid input, and drifts from the real definition unless it is derived from the same types (Duplicate Knowledge).
- Teams typically adopt this for one gnarly module — pricing, parsing, scheduling — and it either stays there or spreads to the pure core. It rarely spreads to orchestration, correctly.
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- DOMAIN-SPECIFICExcellent for pricing, parsing, scheduling, permissions, serialisation and anything algorithmic, where universal statements genuinely exist. Poor for workflow orchestration and UI behaviour, where the "rule" is a sequence of decisions someone made and no general statement is true of it — there, examples are the honest representation.
- LANGUAGE-SPECIFICIn Haskell, Scala or Rust, generators are derivable from types and shrinking is excellent, so the cost per property is close to zero; in JavaScript or Python you usually hand-write generators and shrinking quality varies by library, so the same property costs several times as much to write and produces worse counterexamples. The technique is identical; the price is not.
- SIMPLIFIEDThe shapes listed here — round trip, invariance, commutativity, oracle, metamorphic — are a teaching set, not a taxonomy. Real properties often combine several, and stateful model-based testing is a substantial subject in its own right that this lesson only points at.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — generator design, shrinking strategies, fuzzing, mutation testing and how to run a non-deterministic suite in CI belong there. This lesson only covers what a property says about the design.
- — Programming Languages & Runtime Internals — type-directed generation, and how much of a property a type system can enforce statically instead, is a language-design question with a very wide range of answers.