Prompts and Models Are Deployables
Prompts, model identifiers, tool definitions and policies are versioned production inputs with a blast radius — not settings someone edits live.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
If a prompt change alters behaviour for every user immediately, why is it not treated as a deployment?
Prompts look like text and models look like a setting, so both end up in a place where they can be edited without review, without a test and without a record — while being the inputs that most determine what the system does.
Put the prompt in a database or a prompt-management console so non-engineers can iterate quickly, and put the model name in an environment variable so it is easy to switch. Both are true conveniences and both make sense on the day you set them up.
There is no answer to "what changed?" during an incident, because the change was a text edit with no version, no author and no timestamp anyone can query (Change Correlation).
- There is no answer to "what changed?" during an incident, because the change was a text edit with no version, no author and no timestamp anyone can query (Change Correlation).
- Rollback becomes retyping. The previous prompt is whatever someone can reconstruct, and the previous combination of prompt, model and tools may never be recoverable.
- Staging and production diverge invisibly: the same artifact behaves differently in each because the prompt store differs, and the difference is not in any diff (Environment Drift).
- A model identifier in an environment variable makes the largest behaviour change in the system a config edit that no eval gate sees.
- Nobody can reproduce an old output. Support asks why the agent said something last Tuesday and the inputs that produced it no longer exist.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- What the agent does is a function of five inputs: code, prompts, model, tool definitions and policies. Four of those five are commonly not versioned, which is why agent behaviour feels unpredictable when it is merely untracked.
- Each input has a blast radius, and for most of them it is every request. A system prompt applies to all traffic; a tool description changes tool selection everywhere; a policy change alters what the agent is allowed to do at all.
- Blast radius is what determines the control, not the file type. A one-line text change reaching 100% of users instantly is a bigger change than a hundred-line code change behind a flag (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- Versioning means an identifier that resolves to exactly one content. A prompt referenced as "customer-support-v7" that someone can edit in place is not versioned; a prompt addressed by content hash is.
- Model identifiers vary in how precisely they pin behaviour: some providers offer dated or otherwise specific identifiers that stay fixed, while aliases move as the provider updates them. Whichever you use, record the exact identifier that served each request, because that is the only way to reconstruct behaviour later (The Audit Trail).
- The set has to be pinned together. Prompts are written against a model's behaviour and against a specific set of tool descriptions; a valid combination is the unit, not each part separately.
Five inputs, five blast radii
The middle column is the one that changes how you treat each row. Every input except the code is commonly editable outside the pipeline, and every one of them reaches all traffic at once.
| Input | What changing it does | Default blast radius | Where it usually lives instead |
|---|---|---|---|
| Application code | Alters the plumbing: retries, parsing, routing, side effects | All traffic, gated by the pipeline | Version control — the one input already under discipline |
| System prompt | Alters behaviour, tone, refusals and tool-use tendency on every request | All traffic, immediately | A console or database row, editable in place |
| Model identifier | Changes the reasoning, the failure modes and the cost and latency profile at once | All traffic, immediately | An environment variable, changed without an eval run |
| Tool definitions | Changes when tools are called and with what arguments (Tool Schemas) | All traffic that reaches a tool call | Code, but often edited as "just a description change" |
| Policies and limits | Changes what the agent is permitted to do — allowed tools, approval thresholds, budgets | All traffic, and it is the safety boundary (Tool Permissions and Least Privilege) | Scattered across config, sometimes hardcoded |
One manifest for the whole combination
The combination is the deployable, so the record has to name all of it at once. This is the artifact's declaration of what it will do, and the same values must be readable from a running instance so drift is detectable.
Content hashes rather than friendly names are what make the record load-bearing: a name can be repointed, and a hash cannot.
1{2 "release": "support-agent-2026.03.11-004",3 "artifact": "sha256:4b7e...",4 "code_commit": "9f2c1ab",5 "prompts": {6 "system": { "id": "support/system", "version": 7, "sha256": "1a9c..." },7 "summarizer": { "id": "support/summarize", "version": 3, "sha256": "77de..." }8 },9 "model": {10 "identifier": "<provider-specific pinned identifier>",11 "pinning": "specific",12 "params": { "temperature": 0.2, "max_output_tokens": 1024 }13 },14 "tools": { "version": 12, "sha256": "b304...", "names": ["search_orders", "issue_refund", "escalate"] },15 "policy": { "version": 4, "requires_approval": ["issue_refund"], "max_tool_calls_per_request": 8 },16 "evals": { "dataset": "support-golden@2026-02-28", "run": "ev-88213", "gate": "pass" }17}Note pinning. Recording whether the model identifier is specific or a moving alias is what lets you answer, six weeks later, whether an unexplained behaviour change could have come from the provider. The params belong here too — a temperature change is a behaviour change with no diff anywhere else.
Fast iteration without live editing
The pressure behind live prompt editing is real: the people best at writing prompts are frequently not the people who can open a pull request, and the feedback loop through a full pipeline is long. Refusing the pressure does not remove it.
They edit and save. Behaviour changes for every customer within seconds. No eval runs, no review, no record. When escalation rates move next week, the prompt edit is not in any change feed and the investigation starts from the code.
They edit in a preview environment that runs the same artifact against their draft. An eval run scores the draft against production on the regression set and on escalation-specific cases. The result is attached to a pull request that they can open from the same interface. It ships through the normal path, with a canary.
The loop stays fast where speed matters — writing and seeing the effect — and slows only at the point where behaviour reaches customers, which is the point that needs a record and a reversal. The console is not the problem; the console writing directly to production is.
How to do it properly
Most important first.
- Keep prompts in version control, reviewed like code, and build them into the artifact rather than fetching them from a mutable store at runtime (A Config Change Is a Production Change).
- Pin the most specific model identifier the provider offers. If only a moving alias is available, treat behaviour changes underneath it as an uncontrolled dependency and monitor for them (Dependency Pinning).
- Version tool definitions with the prompt. Tool names, descriptions and schemas are part of what the model reads, and changing a description changes behaviour (Tool Schemas).
- Emit the resolved versions with every request in structured logs and traces: prompt hash, model identifier, tool definition version, policy version (Agent Audit Logs).
- Publish a release manifest naming the whole combination, so "what was running at 14:03" has a single answer (The Release Manifest).
- Give non-engineers a fast, safe path — a preview environment, an eval run on their draft, a pull request from a friendly editor — rather than direct write access to production behaviour. The demand for fast iteration is legitimate; direct production editing is the wrong way to satisfy it.
- Where a prompt must vary per tenant, version the template and treat the per-tenant values as data with their own record. Do not let per-tenant customisation become an untracked prompt fork.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Every one of these inputs applies to all traffic by default. Containment has to be added deliberately: a canary, a cohort flag, or a per-tenant rollout (Canarying a Model or Prompt Change).
What can go wrong
- A prompt registry with mutable versions, where "v3" today is not "v3" from last month and nothing detects it.
- Prompt fetched at runtime with a cache, so instances run different prompts for minutes after a change and behaviour is inconsistent across the fleet.
- Model alias updated by the provider, changing behaviour with no deploy on your side and nothing in your change feed.
- Per-tenant prompt overrides accumulating until no two tenants run the same system and none of the overrides are evaluated.
- Versions logged as the intended values from config rather than the resolved values actually used, so the audit trail records intent rather than fact.
- Prompt versioned but the eval dataset not versioned alongside it, so scores across time are not comparable.
- "Prompts are content, not code." They are executable behaviour specifications. The file format has nothing to do with the blast radius.
- "Version control on prompts slows down the people who write them." It slows down publishing, not writing. Give them a preview environment and an eval run and the loop stays fast.
- "We pin the model, so behaviour is fixed." Pinning removes one source of change. Providers deprecate versions, aliases move, and inference stacks change underneath a stable name; you still need a drift signal.
- "The prompt is in the repository, so we are done." If it is fetched at runtime from a store someone can edit, the repository copy is documentation, not the running system.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can take a request from last month's logs and name the exact prompt content, model identifier and tool definitions that served it.
- A running instance reports its resolved versions, and they match the artifact and the release manifest.
- The last prompt change has a diff, a reviewer, an eval run and a deployment record.
- Staging and production report identical prompt hashes for the same release.
- Rollback is redeploying the previous artifact, which restores the whole combination. Reverting one input alone produces a combination that was never evaluated.
- If the model identifier was an alias that has since moved, rollback restores your code and prompts but not the provider's behaviour — the honest position is that this part is not reversible, which is the argument for pinning specifically (Roll Forward: When Going Back Is the Harder Option).
- Where a prompt change caused harmful output, rolling back stops future occurrences and does nothing about outputs already sent. Those need the same treatment as any other released defect (Telling People What Is Happening).
- Automate version resolution and emission: every request logs the hashes and identifiers actually used, computed at runtime, never copied from a config file by hand.
- Automate detection of drift between the declared manifest and what instances report.
- Automate an alert on a moving model alias: a periodic canary of fixed inputs whose outputs change without a deploy on your side.
- Keep authorship human. Prompt writing is design work; the pipeline should make it reviewable and reversible, not generate it.
- Versioning slows the iteration loop that made a prompt console attractive, and that loop has real value during early development. The answer is a fast preview path, not permanent live editing.
- Building prompts into the artifact means a deploy for every prompt change, which costs pipeline time — and buys you a single, coherent change record.
- Pinning to a specific model identifier means deliberately adopting new versions rather than receiving improvements automatically, which is more work and the entire point.
- Emitting versions on every request adds log volume, and prompt content in logs raises its own handling questions (Secrets in Logs).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThat behaviour inputs must be versioned holds for any model-backed system on any provider. What differs is the precision of the model identifier available to you, which is a provider-specific question worth asking explicitly.
- TOOL-SPECIFICSome prompt-management platforms provide immutable versions, review and rollout controls, in which case they are a delivery system rather than a bypass of one. The test is not where the prompt lives but whether an identifier can ever resolve to different content.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.