Feature Flags: Deploy Is Not Release
Shipping code that is switched off, then turning it on for whom you choose — and the four ways a flag system quietly becomes the least reviewed part of production.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do I get code into production without releasing its behaviour, and what does that decoupling cost over time?
Deployment and release are the same event by default: the moment the artifact lands, the behaviour is live for everyone. That couples an engineering decision — when the code ships — to a product decision — when users see it — and forces both to be made under the same risk.
Add a boolean check around the new behaviour, default it off, and turn it on when we are ready. It is one if statement.
The if is the easy part. The flag has to be evaluated somewhere, which means a configuration source, which means a new dependency in the request path with its own availability and its own failure behaviour.
- The
ifis the easy part. The flag has to be evaluated somewhere, which means a configuration source, which means a new dependency in the request path with its own availability and its own failure behaviour. - What happens when the flag cannot be evaluated is a decision nobody makes deliberately the first time, and it is the decision that determines the blast radius of the flag service being down.
- Flags accumulate. Nobody is assigned to delete them, the old code path stays because deleting it feels risky, and after a year the system has dozens of live branches, most of them permanently in one state (Production Anti-Patterns).
- Two flags are four states; ten flags are more states than anyone has tested. Bugs appear in combinations that no environment ever ran.
- Flipping a flag changes production behaviour instantly with no build, no review, no canary and often no audit trail — which makes it a deploy that skipped the whole pipeline (Change Management).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A flag separates two events that were one: the artifact reaching production (deploy) and the behaviour becoming visible (release). After the separation, each can have its own timing, its own audience and its own reversal (Deployment Is Not Release).
- Evaluation happens at runtime against a targeting rule — everyone, a percentage, a list of accounts, an attribute of the request. The rule is the exposure control, which makes a flag a canary whose unit is the user rather than the request.
- Flags come in kinds with genuinely different lifetimes: a release flag lives weeks and should be deleted; an operational kill switch lives forever and is a reliability feature; an experiment flag lives for the experiment; a permission flag is not really a flag at all, it is authorisation, and it belongs in the permission system (The Agent Kill Switch is the operational form of the second).
- Because a flag is evaluated in the request path, the flag source is a dependency. The robust shape is: fetch rules asynchronously, cache them locally, serve from the last known good copy, and have a compiled-in default for the case where there has never been a copy.
- The default is the real blast-radius control. A kill switch that defaults to "on" when the flag service is unreachable is not a kill switch.
Two events that used to be one
The whole idea in one comparison. On the left, the deploy is the release, so the decision to ship and the decision to expose are made together, under the combined risk of both. On the right they are separate decisions with separate reversals, and each is smaller than the combined one.
merge -> build -> deploy
-> behaviour live for everyone, immediately
-> problem found
-> revert commit -> build -> deploy
-> minutes of full exposure, and a second deploy to undomerge -> build -> deploy (flag off)
-> nothing changes for anyone; code is in production
-> enable for internal users -> 1% -> 25% -> everyone
-> problem found at 1%
-> flip off: seconds, no build, 1% exposedThe right-hand shape reduces both terms of the exposure: fewer users see the problem, and the reversal is a configuration change rather than a pipeline run. It pays for that with a permanent branch in the code and a new dependency in the request path — which is a good trade for a risky feature and a bad one for every trivial change.
The default when the flag cannot be evaluated
Every flag has a fourth state beyond on, off and partially on: unknown. It occurs when the flag source is unreachable, when the rule set has not loaded yet at startup, or when a rename has left the code asking for a key nobody sets.
This is the single highest-leverage decision in a flag system, because it converts a flag-service incident into either a non-event or a company-wide one.
1type FlagKey = 'checkout-v2' | 'kill-recommendations'2 3// Compiled into the binary. The value that is safe if nothing else is available.4const FAIL_SAFE: Record<FlagKey, boolean> = {5 'checkout-v2': false, // unfinished feature: unknown must mean off6 'kill-recommendations': false, // kill switch: unknown must mean "not killed"7}8 9// Refreshed in the background, never fetched in the request path.10let lastKnownGood: Partial<Record<FlagKey, boolean>> = {}11 12export function isEnabled(key: FlagKey, ctx: { accountId: string }): boolean {13 const live = evaluateFromCachedRules(key, ctx) // local, no network14 if (live !== undefined) return live15 const cached = lastKnownGood[key] // survived a restart16 if (cached !== undefined) return cached17 return FAIL_SAFE[key] // never a network call, never a throw18}Notice what is not here: no request-path network call, and no code path that can throw. The reason kill-recommendations defaults to false is that its false state is the normal one — a kill switch whose unknown state kills the feature turns a flag outage into a feature outage, and one whose unknown state ignores the kill is not a kill switch at all. Get that sentence right per flag and write it down.
The four ways flag systems rot
These are not exotic. On a system more than a year old with an unmanaged flag inventory, all four are present simultaneously, and they interact: the stale flag is the one whose off branch rotted, and the wrong default is the one nobody reviewed because the flag was old.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Feature launched; flag never removed | Hundreds of flags evaluated per request, most permanently in one state | No owner and no expiry, and removal is unrewarded work | Expiry at creation, ageing report to the owner, and treat an expired flag as a defect |
| Several flags gating overlapping behaviour | A bug reproducible only for a specific customer | That customer is in a combination of flag states no environment has ever been in | Cap simultaneously live interacting flags; test the combinations that actually exist in production, which you can enumerate from the targeting rules |
| Flag service unreachable at startup | An unfinished feature is live for everyone, or a released feature vanishes | The unknown state falls through to a default nobody chose deliberately | Explicit compiled-in default per flag, plus last-known-good caching; exercise it by blocking the flag source in a lower environment |
| Emergency kill switch flipped for the first time in a year | The off path errors, so the switch makes things worse | Code that has not executed in production for a year is not known to work | Exercise kill switches on a schedule outside incidents — an untested switch is not a control (The Agent Kill Switch) |
| Targeting rule edited during an incident | Behaviour changes for a far wider audience than intended | A flip is a production change with no review, no canary and often no record | Audit every flip with actor and timestamp; annotate the incident timeline with flag changes as well as deploys (Deploys on the Same Timeline as the Symptom) |
| Flag used to gate a paid feature | Users obtain access they have not paid for | A flag is an exposure control, not an authorisation system | Move entitlement into the permission model; keep flags for rollout (Least Privilege in Production for the general principle) |
How to do it properly
Most important first.
- Decide the failure default per flag, and write it next to the flag: what does this evaluate to when the flag service cannot be reached? For a kill switch the safe default is usually "off"; for a long-lived feature already released to everyone it is usually "on".
- Cache the last known good rule set locally and serve from it. A flag service outage should not be an application outage (Validate at Startup, Fail Clearly).
- Give every release flag an owner and an expiry at creation time, and treat an expired flag as a defect with a ticket, not as a tidy-up.
- Delete the losing branch when a flag is retired. A flag removed from the config but left in the code is a code path that will be re-entered by accident.
- Treat a flag flip as a production change: audit who flipped what and when, and put deploy annotations on the timeline so an incident investigation can see it (Deploys on the Same Timeline as the Symptom).
- Ramp flags the way you ramp canaries — a percentage, then more — rather than flipping to everyone, and watch the same signals (Canary Analysis: Compared Against What?).
- Keep the number of simultaneously live, interacting flags small enough that the combinations are enumerable. That is a much smaller number than teams assume.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by the targeting rule while it is correct — a flag enabled for one percent of accounts genuinely affects one percent. It escapes to everyone in three specific ways: a wrong evaluation default when the flag source is unreachable, a targeting rule edited to a broader audience than intended, and a flag whose off branch no longer works so the reversal fails.
What can go wrong
- Stale flags. The feature launched a year ago, the flag is still evaluated on every request, and nobody is sure whether anything still reads the off branch.
- Combinatorial complexity. Behaviour depends on a combination of flags that no test covers and no environment has ever been in.
- Wrong defaults. The flag service is unreachable and the code falls back to a value that enables an unfinished feature for everyone — the flag amplified the blast radius instead of containing it.
- Forgotten code. The off branch has not run in production for a year and has silently rotted; the first time it runs again is during an emergency kill-switch flip, and it does not work.
- Flag evaluation in a hot path adding latency, or a synchronous fetch on every request turning the flag service into a hard dependency.
- Configuration and code disagreeing about what a flag means after a rename, so the running system reads a flag nobody is setting.
- Flags used as permissions, so a routing change accidentally grants access to a feature that was gating a paid tier.
- "Flags let us deploy fearlessly." They let you release gradually. The code is still in production, still executing on every request, and a bug outside the flagged branch ships exactly as it would have.
- "A flag is a rollback." It reverses the behaviour, not the data the behaviour wrote, and it works only if the off path still functions.
- "Feature flags replace canary deploys." A flag controls one behaviour; a canary controls exposure to an entire artifact, including the changes nobody thought to flag — dependency upgrades, runtime changes, refactors.
- "Flags are configuration, so they are low risk." A flag flip changes production behaviour instantly for the audience you targeted, with less review than a one-line code change gets. That is high risk with a friendly interface.
- "We will clean up the flags later." Nothing in this domain gets cleaned up later without an owner and a date.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can list every live flag with its owner, its age, its current targeting and when it was last changed. If that list cannot be produced, the flag system is unmanaged.
- A deliberate exercise of the failure default: with the flag source unreachable, the application starts and serves with the intended values.
- A kill switch has been flipped in production at least once outside an incident, and the off path demonstrably worked.
- Flip the flag back. This is the fastest reversal available anywhere in this domain — no build, no deploy, seconds — and that speed is exactly why the flip needs an audit trail.
- It reverses behaviour, not consequences. Data written while the feature was on remains, and the off path has to tolerate it (Version Coexistence: N and N+1, in Both Directions).
- The reversal depends on the off branch still working. Code that has not executed in months is not known to work, which is the argument for exercising kill switches deliberately.
- If the flag itself is the problem — a bad rule, an unreachable service — the reversal is the compiled-in default, which is why the default deserves as much thought as the feature.
- Automate flag inventory and ageing: a report of flags past their expiry, routed to their owner, is the only thing that reliably prevents flag debt.
- Automate the ramp and the abort for percentage rollouts, exactly as for a canary.
- Do not automate the decision to enable a feature. That is a product judgement, and it is the whole reason for decoupling it from the deploy.
- Every flag is a permanent branch in the code until someone removes it, and the cost is paid in test surface and comprehension, not in runtime.
- Decoupling deploy from release means the code in production is not the behaviour in production, so "what version is running" stops being a complete answer during an incident (Change Correlation).
- A flag service is a new production dependency in the request path of everything, which is a real availability cost against a real containment benefit.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe decoupling of deploy from release, the four failure modes and the failure-default question apply to any implementation, including a boolean read from an environment variable.
- TOOL-SPECIFICHosted flag platforms provide targeting, audit and streaming updates but add a vendor in the request path; an environment variable read at startup provides none of those and requires a deploy to change, which makes it a slower but far simpler reversal. A configuration file in the repository sits in between and gets you review for free. Which one is right depends on how fast you need a flip and how much audit you owe.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.