Business Logic Hiding in a Prompt
A rule that exists only in prompt text is hard to test, hard to enforce, hard to audit, and changes silently when someone edits a sentence or upgrades a model.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
This rule lives in the prompt. Is that a reasonable place for it, or is it logic that escaped the codebase?
A support assistant should escalate to a human when the customer is a paying enterprise account, when the request involves money above a threshold, and when the customer seems upset.
Put all three rules in the prompt. They are stated in one paragraph of plain English that anyone can read and the product manager can edit without a deploy, which sounds like the ideal outcome: the rule is expressed once, in the language the business uses, near the thing that acts on it.
The first two rules are crisp — a tier lookup and a numeric comparison — and giving them to a sampler converts two facts you can compute into two guesses that are usually right.
- The first two rules are crisp — a tier lookup and a numeric comparison — and giving them to a sampler converts two facts you can compute into two guesses that are usually right.
- "Usually right" is the failure. An enterprise conversation is not escalated once in three hundred turns, and because it succeeded the other two hundred and ninety-nine times, nothing detects it until the customer does (Invariant Leaks).
- Someone rewrites the paragraph for tone, and the escalation rule changes as a side effect of a change nobody thought was behavioural. There is no diff a reviewer would flag, because the diff is about wording (Review as Design Feedback — and Why It Arrives Too Late).
- The model is upgraded and the paragraph is interpreted slightly differently. The rule did not change, the code did not change, and the behaviour changed — the specific pathology of putting logic in a component you do not control (The Model Is a Dependency).
- Legal asks for the escalation rule as it stood in March. It is a string in a console with no history, so the answer is "we think it said this" (Audit Logs for Privileged Actions in Security Engineering).
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The prompt is edited by a product manager in a web console, outside code review and outside the deploy pipeline.
- Escalation behaviour is in a customer contract for enterprise accounts, so getting it wrong is a commercial problem, not a quality problem.
- The team cannot enumerate "upset" and should not try — the last attempt was a keyword list that fired on the word "urgent" in an automated signature.
- Every enterprise-account conversation is escalated. This is a contractual guarantee and must hold with no exceptions, including exceptions the customer talks the assistant into.
- A rule that is enforced can be shown to be enforced — as a test that fails when it is violated, not as a sample of outputs where it happened not to be (Enforcing Invariants).
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- Code owns anything with a crisp definition: account tier, thresholds, windows, states, entitlements. These are lookups and comparisons, and they were never judgement.
- The prompt owns what has no crisp definition. Whether a message reads as upset is a genuine judgement and a model is better at it than any rule you would write (Choosing the Model).
- The escalation decision itself belongs to code: a boolean OR over one deterministic input, one deterministic comparison and one model-derived signal. The composition is code even though one of its inputs is not.
- Someone owns the prompt as a versioned artifact — reviewed, diffed, released and rollback-able like any other behaviour-carrying file (Docs Close to Code applies literally: put it in the repository).
- The line is not "AI versus code". It is crisp versus fuzzy, and it is drawn per rule rather than per feature — which is why a feature usually has rules on both sides.
- Model-derived signals cross the boundary as data, with a confidence, and never as a decision.
sentiment: { upset: true, confidence: 0.82 }is an input to a rule; "the model decided to escalate" is the rule having moved (Trust Boundaries). - The threshold applied to that confidence is code, because a threshold is a business decision about tolerance for error, and belongs where it can be changed and tested (Confidence Thresholds in the agentic domain owns how to set it).
The smell, and the case where it is exactly right
This is a smell rather than a rule, which means the useful output is a question and not a verdict. Plenty of prompts contain things that look like rules and are correctly there.
The discriminating test is short enough to apply in review: could you write a test that fails when this rule is violated? For "escalate enterprise accounts" you obviously could, and the fact that no such test exists is the finding. For "match the customer's level of formality" you could not write that test, and any attempt to encode it would be a worse version of what the model already does well.
looks like A prompt containing sentences of the form "if the customer is X, do Y", "never exceed N", "only for accounts on the Z plan" — thresholds, tiers, windows, entitlements and state transitions written in English inside a string that no test asserts against.
suggests A business rule with a crisp definition has been delegated to a component that cannot guarantee it. The rule is now untestable (no assertion can be written), unenforceable (output is sampled), unauditable (no record of which rule fired), and silently mutable (a wording edit or a model upgrade changes it with no diff that looks behavioural).
fix Split the prompt rule by rule. Crisp rules become a predicate in code; fuzzy ones stay and emit a typed signal with a confidence; deterministic code composes the signals into the decision and logs which rule fired.
One paragraph, split along the crisp/fuzzy line
The extraction is less dramatic than it sounds. Two of the three rules were a lookup and a comparison that were being simulated in prose; moving them takes a few lines and makes the prompt better at the job it is actually suited to.
Look at what the model still does in the right-hand version. It reads the message and says how it lands — the thing that has no crisp definition and that no team should be writing regular expressions for. The escalation *decision* is code, and one of its three inputs happens to come from a model.
You are a support assistant. Escalate to a human when: - the customer is on an Enterprise plan - the request involves a refund or credit over $500 - the customer seems upset or frustrated Otherwise, answer directly and be helpful.
// prompt (versioned, in the repo, one job):
// "Classify the customer's tone. Return {upset: boolean, confidence: 0-1}."
function needsHuman(acct: Account, req: Request, tone: ToneSignal): Escalation | null {
if (acct.tier === 'enterprise') return { rule: 'enterprise_tier' }
if (req.moneyInvolved.gt(ESCALATE_ABOVE)) return { rule: 'money_threshold' }
if (tone.upset && tone.confidence >= UPSET_THRESHOLD) return { rule: 'tone_signal' }
return null
}Two of the three rules became provable: a test asserts that an enterprise account escalates, and it fails if anyone breaks it. The third stayed fuzzy because it *is* fuzzy — but the threshold that turns it into a decision is now a constant someone owns, and the rule field means an auditor can ask which rule fired and get an answer that is an enum rather than a paragraph. The prompt also got better at its remaining job, because it now has one.
Deciding, rule by rule
Applied per feature this becomes a slogan; applied per rule it becomes a design tool. Most features have rules in three or four of the categories below, and the mistake is almost always uniform treatment — putting everything in the prompt because the fuzzy part had to be there, or extracting everything because the crisp part should not have been.
Could you write a test that fails when this rule is violated — and does the rule have to hold every time?
when Refund limits, plan gating, contractual escalation, anything an auditor could ask about.
cost Move it to code. You pay a deploy for every change and lose the ability to tune it from a console, and you get an assertion instead of an estimate (Where Invariants Live).
when "Prefer suggesting self-service before opening a ticket."
cost The prompt is fine. Nothing breaks when it is occasionally not followed, and encoding it in code buys rigidity for a rule with no guarantee to make.
when "Is this customer upset?", "do these two reports describe the same bug?"
cost Keep it in the prompt and emit a typed signal. You pay for a labelled evaluation set and a threshold that needs periodic recalibration — cheaper than the keyword list you would otherwise write, and much better.
when "Escalate if the customer seems likely to churn", where escalation costs an hour of a specialist.
cost Split it: the model emits the signal, code applies a threshold, and a human confirms above a value. The threshold is the design decision and it belongs in code where it can be moved and tested (Approval Gates and Risk Classes in the agentic domain).
when "Handle abusive messages appropriately."
cost That is a requirements problem wearing a prompt costume, and no placement fixes it. Go and define it before deciding where it lives (The Requirements Nobody States).
How to build it
Most important first.
- Read the prompt and list every rule in it. Then, per rule, ask the only question that matters: could I write a test that fails when this is violated? If yes, it is code that has escaped into prose.
- Move the crisp rules out and leave the fuzzy ones in. The prompt gets shorter and better at what remains, which is usually a quality improvement as well as a correctness one.
- Have the model emit a signal, not a verdict — a typed classification with a confidence — and let deterministic code compose the signals into the decision (Structured Outputs in the agentic domain).
- Put the prompt in version control next to the adapter that uses it, and treat an edit as a deploy. A behaviour-carrying artifact that bypasses review and rollback is a hole in your release process regardless of what it is written in (Prompts and Models Are Deployables in DevOps).
- Write the crisp rules as one readable predicate that a non-engineer can be walked through. The reason the prompt felt right was that it was legible to the business, and losing that legibility is a real cost you should not pay by accident (Ubiquitous Language).
- Log which rule fired. "Escalated: enterprise_tier" is auditable; "escalated because the assistant judged it appropriate" is not (Stable Identifiers).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Threshold moves from 500 to 250: one constant and one test under the split design. In the prompt, it is a text edit with no test, and the honest verification is a sampling exercise that cannot prove the old behaviour is gone.
- A model upgrade under the split costs re-calibrating one sentiment classifier against a labelled set — a bounded task with a number at the end. With the rules in the prompt it costs re-validating escalation, tone, formatting and every other behaviour that paragraph was carrying, with no list of what those behaviours were.
- Adding a fourth rule — escalate on a repeat contact within 24 hours — is a clause in a predicate and a test. In the prompt it is another sentence competing for attention with the existing ones, and adding it can degrade the others in ways no diff shows.
- The cost that goes *up*: a change to a genuinely fuzzy behaviour now requires a prompt edit plus a threshold review plus an eval run, where before it was one sentence. The split makes crisp rules cheap and makes fuzzy ones slightly more expensive to tune.
- Extracting the crisp rules means the business can no longer change escalation without a deploy. That is the point and it is also a real loss of speed, and teams that value the speed more are not obviously wrong for a low-stakes feature.
- Two artifacts now carry the behaviour — a predicate and a prompt — and a reader has to look at both. One paragraph was genuinely easier to read.
- Composing signals in code means you lose the model's ability to weigh the whole situation at once. There are cases where "technically below the threshold but clearly a serious problem" was handled well end-to-end and is now handled mechanically.
What can go wrong
- The rules are extracted into code and also left in the prompt "so the model has context", so the threshold now exists twice and drifts on the first change (Duplicate Knowledge).
- Everything is extracted, including the fuzzy part, and the sentiment rule becomes a keyword list that fires on "urgent" in a signature. This is the opposite mistake and it is common in teams that learn this lesson too enthusiastically.
- The confidence threshold is set once from a demo and never revisited, so as the model changes underneath it, the calibration silently drifts (Revisit Triggers).
- The prompt is version controlled but the *model id* is not, so a rollback restores the text and not the behaviour.
- The escalation rule depends on the account service for tier and on the model for sentiment. One of those dependencies is reliable and one is not, and the design keeps them distinguishable at the point of use.
- The prompt depends on the model, which means it is a coupled pair: a prompt revision is only meaningful against a model version, and the two must be released and recorded together.
- "So nothing belongs in the prompt." The fuzzy rules belong there and would be worse as code — that is not a concession, it is the reason the model is in the system at all. A team that extracts sentiment detection into keyword matching has made the product worse and called it engineering.
- "Version controlling the prompt fixes it." It fixes auditability and rollback, which are two of the four problems. It does nothing for enforceability: a rule in a reviewed prompt is still a tendency (Enforcing Invariants).
- "Test the prompt harder." More sampling raises confidence and never reaches a guarantee. The distinction between evidence and proof is the whole lesson, and no amount of evaluation crosses it (Evaluating Agents: Testing Probabilistic Systems in the agentic domain is clear about what evals can and cannot establish).
- "The model is smart enough now." Capability is not the issue. A component whose output is sampled cannot make a guarantee at any capability level, and the rules being discussed here are ones a comparison operator gets right every time (The Model Is Not the Authorization Layer in Security Engineering makes the same point about authorization).
- prompt-logic-smell
- duplicate-knowledge
- invariant-leaks
Testing it, and how it ages
- Unit test the predicate exhaustively: enterprise plus calm, free plus furious, threshold minus one cent. Fast, deterministic, and readable by the person who owns the contract (What a Unit Is).
- Test the sentiment classifier against a labelled set with an accuracy threshold, separately, on its own schedule. It is the only statistically-tested thing left (Golden Datasets in the agentic domain).
- A property test that no combination of model output can produce a non-escalated enterprise conversation. That is the contractual invariant, and it should be impossible rather than unlikely (Property-Based Testing).
- A characterization test captured *before* extracting the rules from the prompt, so "we did not change the behaviour we meant to keep" is checkable (Characterization Tests).
- The prompt shrinks as rules are extracted, and a short prompt with one job is easier to change and easier to evaluate. This tends to be self-reinforcing in a healthy codebase.
- As models improve, the fuzzy side handles more, and rules genuinely migrate from code to prompt. That direction is legitimate — what makes it safe is that it is a reviewed change with an eval, not a drift.
- The pressure that eventually breaks this design is rule volume: at fifty crisp rules the predicate becomes a policy engine, and at that point the question stops being prompt-versus-code and becomes whether the rules deserve their own representation (Strategy).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThe four costs — untestable, unenforceable, unauditable, silently mutable — follow from the artifact being prose interpreted by a sampler, so they apply to any model, any vendor and any framework. They are not claims about current model quality.
- DOMAIN-SPECIFICThe strength of the argument scales with the cost of being wrong. For a contractual escalation or a refund cap, a tendency is unacceptable; for choosing which of five help articles to link, a tendency is completely fine and extracting it into rules would be over-engineering.
- CONTESTEDThe strongest opposing case: prompts are the fastest-changing part of these systems, and moving rules into code makes every iteration a deploy — slowing the loop that actually improves the product, and hard-coding a decomposition that a more capable model handles better holistically. Practitioners who ship consumer AI products at speed hold this seriously. The disagreement narrows to a testable question: does this rule need to be *guaranteed*? Where the answer is yes, both sides agree; where it is no, the fast-iteration position is often right.
- SIMPLIFIEDThe three-rule example treats sentiment as a single boolean signal. Real classification returns a distribution over several dimensions and the threshold question is correspondingly harder; the structural argument is unaffected, but the calibration work is larger than shown here.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — what a labelled evaluation set can and cannot establish about a rule, and how a threshold on a distribution becomes a release gate, is theirs.