Artifact Retention
Which artifacts you can delete, why "keep the last N" deletes the one you needed, and why retention policy is part of the rollback plan.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Which stored artifacts are safe to delete, and what breaks the first time you delete the wrong one?
Artifact stores grow without bound. Every policy that bounds them is a rule about which past you are willing to lose, and most of those rules are written by whoever was looking at the storage bill.
Storage is cheap — keep everything. Or, once the bill arrives: keep the most recent N images per repository and delete the rest. Old builds are not useful.
"Keep the last N" is ordered by push time, and deployment is not. An environment that moves slowly — a customer-managed installation, a rarely-updated batch job, a paused canary — is running something that fell off the list weeks ago.
- "Keep the last N" is ordered by push time, and deployment is not. An environment that moves slowly — a customer-managed installation, a rarely-updated batch job, a paused canary — is running something that fell off the list weeks ago.
- The rollback target is by definition not the newest artifact. A policy tuned to keep only recent builds is precisely a policy that deletes rollback targets, and it does it silently.
- In container registries, deleting a tag is not deleting the artifact. The manifest may persist untagged and remain pullable by digest until garbage collection runs, so teams both fail to reclaim the space they wanted and lose the reference they were using.
- The inverse also happens: a policy that keeps "tagged" artifacts and prunes untagged ones deletes exactly the artifacts a digest-based deploy references, because a promoted artifact may carry no tag at all.
- Nothing fails at deletion time. The failure surfaces when a node is replaced or a rollback is attempted, which is hours or months later, and by then the deletion is not a suspect.
- Keeping everything has its own costs beyond storage: listing and scanning surfaces grow, vulnerability reports fill with artifacts nobody runs, and finding the artifact you want takes longer every month.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Retention is a rule evaluated over artifact metadata: age, tag pattern, repository, and — the one that matters — reachability. An artifact referenced by a running deployment, a release record, or the immediate rollback target of either is not garbage, whatever its age.
- Registries do not know your reachability. They know tags, ages and counts. The gap between what the registry can see and what your platform is running is where every accidental deletion lives.
- Container registry deletion is two-phase in most implementations: removing a tag or manifest makes it unreferenced, and a separate garbage collection pass removes blobs no manifest references any more. Space is not reclaimed at delete time, and a shared base layer survives as long as any image uses it.
- A running container does not re-pull, so deletion has no immediate effect. The consequence appears at the next pull — a rescheduled workload, a replaced node, an autoscaling event, a rollback — which is why the incident is always detached from the cause.
- Some artifacts have a retention requirement rather than a retention budget: anything that reached production may be needed months later to reproduce a security finding or reconstruct an incident (The Audit Trail).
Classes of artifact, and what each one is worth keeping for
A single retention rule across all artifacts is what produces both problems at once — too much storage and a missing rollback target. Classifying first makes the policy obvious.
The volume is concentrated in the top rows and the value is concentrated in the bottom ones, which is a convenient shape: the aggressive policy and the conservative policy apply to different populations.
| Class | Typical volume | Why you would want it later | Retention shape |
|---|---|---|---|
| Pull-request / branch build | Highest | Almost never | Days, or until the branch is merged or closed |
| Main-branch CI build not promoted | High | Bisecting a regression | Short window, count-bounded |
| Reached a pre-production environment | Medium | Reproducing a test failure | Weeks |
| Reached production | Low | Rollback, incident reconstruction, security forensics | Long window; never count-bounded |
| Currently deployed anywhere | Lowest | It is running | Never delete, regardless of age |
| Immediate rollback target | Lowest | It is the recovery plan | Never delete while its successor is deployed |
Choosing the policy shape
Every option below is in use somewhere and defensible. What is not defensible is choosing one without knowing which artifacts it would have deleted last month.
What decides whether an artifact is deleted?
when A high-volume repository of CI-only builds where nothing lives long.
cost Ordered by push time, which has no relationship to what is deployed. Deletes rollback targets for anything slow-moving.
when Predictable release cadence and a known maximum time between deploys.
cost An environment that has not deployed in longer than the window loses its artifact, and nothing warns you.
when Tags reliably encode intent — release tags kept, branch tags expired.
cost Fails completely for digest-based promotion, where the artifact you must keep may carry no tag at all.
when You can query the platform for what is deployed and what each environment's previous release was.
cost Requires that integration to be built and kept correct; a stale reachability source is worse than none, because it is trusted.
when The usual right answer: never delete what is reachable, then apply age tiers per class to everything else.
cost More moving parts, and the exclusion query becomes load-bearing infrastructure of its own.
when Early, low volume, and storage genuinely is not a constraint yet.
cost Deferred, not avoided. Listing, scanning and search degrade, and the eventual cleanup is done under bill pressure by someone in a hurry.
Deletions that surface much later
The unifying property of this table is delay. In every row, the delete succeeds, the system stays healthy, and the failure arrives at the next event that requires a pull.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| GC removed an untagged manifest | A healthy deployment cannot be rescheduled on a new node | Digest-based deploy of an artifact whose tag was removed by policy | Exclude reachable digests; re-push the artifact if you still have it, otherwise roll forward |
| Keep-last-N ran | Rollback command fails with a manifest-not-found error mid-incident | The rollback target aged out by push count | Roll forward with a fix; afterwards, make rollback targets an exclusion (Roll Forward: When Going Back Is the Harder Option) |
| Base image pruned from a mirror | Builds fail; deployments are fine | Retention applied to a mirror repository nobody classified | Treat mirrored base images as production inputs with their own retention (Dependency Pinning) |
| Retention applied to fix a scan report | Vulnerability count drops; the vulnerable artifact still runs | Deleting the record rather than the deployment | Redeploy from a patched artifact first; delete afterwards, if at all (Scanning, and Why a Finding Is Not a Risk) |
| Soft-delete window elapsed | An artifact thought recoverable is gone | The window was shorter than the time it took to notice | Match the window to your detection time, which is measured in scheduling events rather than hours |
| Policy changed to cut storage cost | Nothing, for weeks; then a rollback fails | A cost change with an unexamined recovery consequence | Treat retention changes as changes to the rollback plan and review them as such (Change Management) |
How to do it properly
Most important first.
- Classify before you set a policy. CI builds that never left the pipeline, artifacts that reached a pre-production environment, and artifacts that reached production are three different things with three different lifetimes.
- Make "currently deployed anywhere" and "immediate rollback target of anything deployed" hard exclusions the policy cannot override. Feed that list from the platform, not from tags.
- Retain everything that reached production for a long window — long enough to cover incident reconstruction and any compliance requirement you actually have.
- Expire aggressively at the other end: per-branch and per-pull-request builds are the bulk of the volume and the least valuable (Preview Environments).
- Make deletion reversible where the store allows it — a soft-delete window, or an archive tier — so a mistake costs a restore rather than a rebuild.
- Run the policy in dry-run mode first and read what it would delete. This is the single step that catches the reachability bug, and it takes one afternoon.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains it at the time of deletion, because nothing observable happens. What contains the eventual impact is whether anything needs to pull: a stable fleet is unaffected until it is not, which makes this the module's longest-fused failure.
What can go wrong
- Garbage collection removes an untagged manifest that a deployment references by digest. Everything keeps running until a pod is rescheduled, and then it cannot start (Tags Versus Digests).
- A rollback fails because the previous artifact is gone, converting a two-minute recovery into a build-and-verify cycle during an incident (Rollback: Only Useful If It Is Actually Safe).
- Base image layers deleted from a mirror, breaking builds rather than deployments, with an error that points at your Dockerfile.
- Retention applied per repository while a service's artifacts are split across repositories, so the policy is stricter in practice than anyone intended.
- The mitigation failing: an exclusion list built from a snapshot of deployments taken once, which goes stale and starts protecting the wrong artifacts.
- Deleting to reduce a vulnerability report rather than to reclaim space — the finding disappears, the artifact keeps running somewhere, and the report is now wrong (Scanning, and Why a Finding Is Not a Risk).
- "Deleting a tag frees the space." In most container registries it removes a name. Blobs go when garbage collection runs and no manifest references them.
- "We keep the last twenty, that is plenty." Twenty of what? Twenty pushes can be a single afternoon, and the artifact serving a slow-moving environment is not among them.
- "It is fine, the containers are already running." Running containers do not re-pull. The failure waits for the next scheduling event, which may be during the incident you are trying to fix.
- "Retention is a cost problem." It is a recovery problem with a cost dimension. The policy is part of your rollback plan whether or not anyone wrote it down there.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A dry run of the policy lists nothing that any environment currently references.
- A rollback to the previous production release succeeds from a fresh node, tested rather than assumed (Restore Drills is the same idea applied to data).
- Storage growth is bounded and the bound is a decision someone made, not an accident.
- For any release in the retention window, the artifact it names can still be pulled by digest.
- Deletion is the one operation in this module that is genuinely irreversible unless the store offers a soft-delete or archive tier. Rebuilding from the same commit produces a different artifact and does not restore the identity (Reproducible Builds narrows the difference and does not eliminate it).
- This asymmetry is the argument for making the policy conservative on the production side and aggressive on the CI side: the cheap deletions are the ones you can afford to be wrong about.
- Automate the aggressive end completely: branch builds, pull-request builds and untagged CI output on a short clock.
- Automate the reachability query — what is deployed, what is the rollback target — and feed it into the policy as an exclusion, refreshed on every run rather than once.
- Do not automate deletion of production-reaching artifacts. That should be a periodic review a human signs off, because the failure mode is unrecoverable and the saving is small.
- Long retention costs storage and slows down listing, scanning and search, and it buys the ability to reconstruct the past.
- Aggressive retention saves money and converts every unusual rollback into a rebuild.
- Soft-delete windows give you a mistake budget and mean the space is not reclaimed when you expected it, which makes storage forecasting less predictable.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICGarbage collection semantics are the thing that differs most between registries: whether untagged manifests are collected automatically or only on an explicit run, whether deletion is soft or immediate, whether policies are expressed by tag pattern, age or count, and whether blob reclamation requires the registry to be read-only during the pass. The reachability principle is general; every mechanism implementing it is not.
- ORG-SPECIFICHow long an artifact that reached production must be retained is usually a compliance or audit answer rather than an engineering one, and it varies from "until the next release" to "years". Ask rather than assume, because the two ends of that range imply very different storage decisions.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — a rollback path that has never been exercised from a fresh node is a plan, not a capability.