Duplicate Knowledge
Two identical blocks may not be the same concept, and two blocks that look nothing alike may encode the same rule. Textual similarity is the wrong test, and it is the one everybody uses.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
These two blocks are identical. Should they be one — and what about the two that are completely different but always change together?
Loyalty eligibility changes from "spent over 500 in a year" to "spent over 500 in a rolling twelve months". The rule turns out to exist in a SQL report, a nightly job, a React badge and a validation check — in four completely different shapes.
Duplication is bad, so find the repeated code and extract it. A linter can find identical blocks, so start there — it is objective and it scales.
The linter finds the wrong things in both directions. It flags two validators that both check a string is non-empty — same lines, unrelated rules — and it cannot see the four eligibility implementations at all, because they share no tokens.
- The linter finds the wrong things in both directions. It flags two validators that both check a string is non-empty — same lines, unrelated rules — and it cannot see the four eligibility implementations at all, because they share no tokens.
- Merging look-alikes creates a shared thing that must satisfy two independent reasons to change, and the next requirement pulls it in two directions. The result is a function with a flag, then two flags, then a strategy parameter (Speculative Generality).
- The damage is asymmetric and that asymmetry is the whole lesson: duplication is visible and locally annoying, a wrong abstraction is invisible and globally expensive. You can see the first in a diff; you find the second when a change to tag validation breaks account signup.
- Meanwhile the genuine duplicate knowledge — the four eligibility rules — keeps drifting, because nothing anywhere says they are the same thing (What Makes Software Hard to Change).
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The SQL report is owned by the analytics team and runs against a replica (Read Replicas From the Application).
- The React badge cannot call the backend synchronously on render, so it has a copy of the rule in TypeScript.
- The four implementations already disagree in edge cases, and nobody knows which one is correct.
- A customer shown as eligible must be eligible when they try to redeem. Today that is a coincidence maintained by care.
- When the rule changes, all four must change together or the system contradicts itself in public.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- One place must own the eligibility rule as a *decision*, and the other three must derive from it rather than restate it.
- Where a copy is unavoidable — the SQL report on a replica, the client-side badge — something must own generating or verifying that copy, so the drift is detectable rather than discovered.
- Nobody owns "removing duplication" as an activity. The owner is of the knowledge, not of the text (DRY: Knowledge, Not Lines).
- The boundary goes around the decision, expressed so that all four consumers can use it: a predicate over facts, not a SQL fragment and not a React hook.
- Where the boundary genuinely cannot be crossed — a database engine that must evaluate the rule in a query — the seam becomes a generated artefact or a contract test, not a second hand-written implementation (Contract Tests).
- Two things that look identical and answer to different owners must stay on opposite sides of a boundary, even at the cost of visible duplication.
The smell, and the two ways to get it wrong
This entry has an unusual shape, because the smell has a false-positive twin. What you are looking for is not repetition — it is a decision with more than one authoritative representation. Those two sets overlap much less than the catalogue suggests.
looks like The same business rule expressed in a SQL WHERE, a validator, a report and a UI badge — usually in different words, sometimes in different languages. Or, in the false-positive direction, two identical helper functions that a similarity tool has flagged.
suggests A decision has no single home. Every consumer reconstructed it, so they will drift, and the drift will be discovered by a user noticing a contradiction rather than by a test.
fix Ask whether the two must change together. If yes, give the decision one home and derive or verify the copies. If no, leave them alone no matter how similar the text is, and do not let a similarity tool make the decision.
// shared/validate.ts export const isValidLength = (s: string) => s.length >= 3 && s.length <= 50 // callers: username, product tag // "usernames may be 2 chars" ships; // the search index rejects short tags // at 3am. Nobody links the two.
-- report.sql (analytics team) WHERE spend_12m > 500 // badge.tsx if (spendThisYear > 500) show(<Gold/>) // Nothing links these. "Year" and // "rolling 12 months" already disagree // and have for eight months.
Both are failures of the same test, applied in opposite directions. The left pair must not change together and was merged; the right pair must change together and was not. Any rule based on textual similarity gets both of these wrong, which is why the test has to be "must these change together?" and why answering it requires knowing the domain rather than reading the code (DRY: Knowledge, Not Lines).
Pricing the rule change
The four-implementation case is worth pricing carefully, because the "after" design is not the clean one people expect. Two of the copies cannot be removed — the analytics team needs SQL against a replica, and the badge cannot make a network call on render. The design that wins does not eliminate copies; it makes disagreement between them a build failure.
Loyalty eligibility changes from "spent over 500 this calendar year" to "spent over 500 in a rolling twelve months", with the same threshold and a new time window.
Four edits by three teams, no list of where the implementations are, and no way to know they now agree. The existing disagreement in edge cases is invisible to all four test suites, because each one tests its own implementation against its own expectations.
One edit and a regenerated artefact. The corpus test is the real change: a divergence between the rule and the SQL now fails the build instead of being found by a customer.
Deciding, without a linter's help
In practice the judgement is made in a few seconds during review, so it needs to be a small number of questions. These are ordered so that the cheapest and most decisive one comes first.
If this decision changes, must both places change?
when Two services in one codebase both compute the discount.
cost Merge into one owner. Cheapest case there is, and the only one where "remove the duplication" is unambiguously right.
when A SQL predicate on a replica; a rule the client must evaluate offline.
cost Keep the copy, generate or verify it, and add a shared corpus test. You are buying detection, not elimination (Contract Tests).
when Username length and tag length; two "is non-empty" checks.
cost Leave both. Accept that a similarity tool will keep flagging it and write down why, once, so the argument is not had again (Architecture Decision Records).
when Two rules that have been identical for a year and might be one concept.
cost Wait for the third occurrence. Duplication is cheap to keep and cheap to merge; a wrong merge is expensive and silent (The Rule of Three).
when The four eligibility implementations.
cost Stop and find out which one is correct before designing anything. You have a live bug, and the design work is the second problem (Characterization Tests).
How to build it
Most important first.
- Apply the only test that works: when this decision changes, must both places change too? If yes, one piece of knowledge. If no, leave them alone however similar they look (DRY: Knowledge, Not Lines).
- Find same-knowledge duplication by following requirements, not by scanning text. Take the last three changes of this kind and list what each touched (Shotgun Surgery).
- Give the rule one home in the language that can express it most completely, and derive the rest — generate the SQL predicate, ship the rule to the client, or verify the copies with a shared test corpus.
- When a copy must exist, make disagreement loud: a test that runs the same cases through both implementations and fails when they diverge (Contract Tests).
- When you cannot tell whether two blocks are the same knowledge, wait. Duplication is cheap to keep and cheap to merge later; an incorrect merge is expensive to detect and expensive to unwind (The Rule of Three).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Before: changing eligibility costs four implementations in four languages, owned by three teams, with no list of where they are. The expensive part is establishing that you found them all, and you cannot — the fourth was found because a customer complained.
- Before, compounding: every consumer added makes the next rule change more expensive, so the cost grows with the popularity of the concept, and it grows silently because each addition is one small copy.
- After: changing eligibility costs one edit to the rule, a regenerated SQL predicate, and a shared corpus test that fails loudly if any consumer disagrees. The number of consumers stops appearing in the cost.
- What a wrong merge costs, for contrast: changing a username length rule and silently changing tag validation, discovered weeks later by a search-indexing bug. That cost is unbounded because it is not attributed to the change that caused it.
- Deriving copies costs build machinery, and build machinery is a dependency with its own failure modes (What a Build System Actually Is).
- A single owner for a rule that four consumers use is a coordination point, and the client-side consumer now needs a release to pick up a rule change that used to be a one-line edit.
- Waiting until you are sure means living with duplication you know about, which is uncomfortable and looks like negligence to anyone applying DRY as a line-count rule.
What can go wrong
- Look-alikes are merged and the shared function accumulates parameters until it is a small interpreter for its own callers.
- Same-knowledge copies are centralised in one language and the other three consumers keep their versions "for now", so the system has a canonical rule and three copies that no longer even claim to match.
- The consistency test is written against the happy path, so the copies agree on the cases that were never going to differ.
- The mitigation fails: the "must both change together" test depends on predicting what the business will ask for, and when the business splits a rule that was genuinely one — consumer and business customers diverging — the correct past merge becomes a present obstacle.
- Consumers depend on the rule owner rather than on each other, which is the direction that lets the rule change without a coordinated release (Dependency Direction).
- A generated copy creates a build-time dependency, which is visible and testable; a hand-written copy creates a dependency on someone remembering, which is neither.
- "DRY means never write the same lines twice." It means every piece of *knowledge* has a single authoritative representation. The original formulation was about knowledge, and the line-count reading is how it became harmful (DRY: Knowledge, Not Lines).
- "So duplication is fine." Duplicated knowledge is the most expensive property a codebase can have — it is what makes the tenth change cost ten times the first (What Makes Software Hard to Change).
- "A code-similarity tool can find this." It finds textual repetition, which is neither necessary nor sufficient. The four eligibility rules share no tokens (What to Automate Out of Review).
- "The client copy is just a performance optimisation." It is a second implementation of a business rule, and it will disagree. Treat it as a copy that must be verified, not as a detail (Deliberate Debt).
- duplicate-knowledge
- shotgun-surgery
- divergent-change
Testing it, and how it ages
- A shared corpus of cases, run against every implementation of the rule, including the SQL one. This is the single highest-value test in the system and it is usually missing (Contract Tests).
- Test the rule itself as pure logic over facts, with no database and no clock injected implicitly (Time as a Dependency).
- Before merging two look-alikes, write the test that distinguishes them. If you cannot write one, that is evidence they might be the same knowledge — not proof, but the best evidence available.
- Same-knowledge duplication is created by convenience under deadline and by boundaries the rule could not cross. Both are ordinary and neither is a failure of care.
- One rule genuinely splitting into two is common — a policy that applied to all customers starts differing by segment — and a codebase that merged aggressively finds this painful, which is the honest cost of merging.
- Generated copies age well; hand-written ones drift at a rate proportional to how many teams touch them.
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThe distinction between duplicated text and duplicated knowledge holds in every language and paradigm; what varies is which mechanism can remove a copy — a shared module, a generated artefact, a macro or a compile-time check.
- CONTESTEDThe strongest opposing position, and it is a serious one: "duplication is far cheaper than the wrong abstraction", argued by people who have spent years unwinding shared code whose callers diverged. Their claim is not that duplicate knowledge is harmless but that engineers are systematically bad at telling the two cases apart in advance, so the expected-value play is to wait longer than feels right. This lesson agrees with the diagnosis and stops short of the conclusion: the four-language eligibility rule is not a case where waiting helps.
- LANGUAGE-SPECIFICWhen a rule must be evaluated inside a database, no shared function can span the boundary, and the honest options are code generation or a shared test corpus. In a system where the same language runs on both sides — TypeScript on client and server, for instance — the copy can genuinely be removed, which makes the advice much easier to follow there than in a polyglot stack.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — a shared corpus run against several implementations of one rule is the testing technique this lesson depends on, and its coverage and maintenance properties are developed there.