Problem says X → think Y
The searchable index of this domain. The left column is what the problem sounds like when someone says it to you in a channel at 3am; the right column is the thing to think before you start typing commands.
86 of 86 rows
| The problem says | Think |
|---|---|
| The build produces a different result on my machine than in CI | The build is not reproducible: unpinned dependencies, host toolchain, timestamps or ambient environment are leaking into the output. Pin the inputs and build in a controlled environment.Reproducible Builds → |
| The thing running in staging is not the thing running in production, even though it is the same commit | You are rebuilding per environment. Build once, promote the same artifact, and let only configuration vary.Build Once, Deploy Many → |
| CI takes forty minutes so people push and walk away | CI has stopped being a feedback system. Order checks by how fast they can say no, parallelise, and cache what is genuinely input-addressed.CI Is a Feedback System → |
| A test fails, is re-run, and passes | Flakiness is a signal about concurrency, time or shared state — not a reason to add a retry. Quarantine loudly, fix or delete; a retried test teaches the team that red means nothing.Flaky Tests → |
| The CI cache made a build wrong rather than fast | The cache key does not fully describe the inputs. A cache is only safe when the key is a hash of everything that can change the output.Caching in CI → |
| The pipeline runs every job on every commit, including ones nothing touched | You need a dependency graph over the repository, not a list of jobs. Affected-target selection is the difference between a monorepo that scales and one that does not.The CI Dependency Graph → |
| Nobody can tell whether a red pipeline is a real failure or infrastructure | Failure triage needs classification at the source: test failure, flake, infrastructure, or dependency outage. Without it every red build costs the same attention.Triaging a CI Failure → |
| The tag we deployed points at a different image than it did last week | Tags are mutable pointers. Deploy by digest; use tags for humans and digests for machines.Tags Versus Digests → |
| We cannot say which source tree an image in the registry came from | The artifact has no provenance. Attest the build: which builder, which source revision, which inputs, signed at build time.Build Provenance → |
| The registry bill grows every week and nobody deletes anything | Retention is a policy question with a rollback constraint: you must keep every artifact you might need to roll back to, and no more.Artifact Retention → |
| A transitive dependency changed under us without any commit of ours | Version ranges resolve at build time. Lock files and pinned digests turn "latest compatible" into an input you can reason about.Dependency Pinning → |
| A scanner reports two hundred vulnerabilities in an image | A finding is not a risk until you know whether the vulnerable path is reachable, exposed, and exploitable in your configuration. Triage with context or the queue trains people to ignore it.Scanning, and Why a Finding Is Not a Risk → |
| We need to know every component in a shipped artifact after a disclosure | That is an SBOM question, and it is only answerable if the bill of materials was generated at build time by the builder.Software Bill of Materials → |
| A dependency of ours was compromised upstream | Everything between a dependency and a running artifact is attack surface: pinning, provenance, signing, and a trusted builder that untrusted code cannot influence.The Delivery Chain as Attack Surface → |
| The image is several gigabytes and pulls dominate deploy time | Image size is a deploy-latency and scale-out-latency problem before it is a storage problem. Multi-stage builds and layer ordering, not compression tricks.What Image Size Actually Costs → |
| Staging passed and production broke | Pre-production evidence has limits: traffic shape, data volume, dependency behaviour and configuration all differ. Parity narrows the gap; it never closes it. More environments is not more safety.Parity That Is Worth Paying For → |
| The service started fine and failed two hours later on a missing config value | Configuration is validated lazily. Validate the whole config at startup and refuse to start, so the failure is a failed deploy instead of a 3am page.Validate at Startup, Fail Clearly → |
| An outage was caused by a change, but there was no deploy | Config is a deployable with its own blast radius. It needs review, versioning, staged rollout and rollback exactly like code.A Config Change Is a Production Change → |
| A credential is in the repository or baked into the image | Secrets belong in a secret manager, injected at runtime; better still, replace the long-lived credential with workload identity so there is nothing to leak.Secret Managers and What They Actually Give You → |
| Rotating a secret took the service down | Rotation only works if the application can hold two valid credentials at once. Overlap windows, then revoke — the same expand/contract shape as a schema change.Rotation That Applications Survive → |
| A CI job needs production credentials to deploy | Short-lived federated identity scoped to the job, not a static key in a variable that every fork-triggered build can reach.Secrets in CI → |
| We copied a production database into staging so the tests would be realistic | You have just extended production data handling obligations to a lower-trust environment. Subset, mask or synthesise instead.Production Data in Lower Environments → |
| Production does not match what the infrastructure code says | Drift. Find out what changed it, reconcile deliberately rather than blindly, and close the path that allowed the out-of-band change.Drift → |
| Someone fixed it by clicking in the console at 2am | The fix was correct and the mechanism is a problem: that change is now invisible, unreviewed and will be reverted by the next apply. Capture it back into code and decide whether the console path should still exist.Manual Production Changes → |
| We need to be able to rebuild this environment from nothing | That is the reproducibility case for infrastructure as code — reviewable, diffable, and standable-up from a repository.Infrastructure as Code → |
| The plan says it will destroy and recreate the database | A rename or a changed immutable field became a replacement. Read plans for deletions first, and put a lifecycle guard on anything that holds state.Destructive Changes: What a Rename Really Does → |
| Two people applied at the same time and the state file is now wrong | State is shared mutable infrastructure. Remote state, locking, and one path to apply — usually CI, not laptops.State → |
| We need to roll out a change without exposing everyone to it | Pick the strategy that matches the risk: canary for a slice of live traffic, blue/green for an instant switch with a spare environment, flags for per-user control.Deployment Strategies → |
| During the rollout, old and new versions are both running | That is always true of a rolling deploy, so every change must be backward compatible with the version it is replacing — in the API, the queue payload and the schema.Version Coexistence: N and N+1, in Both Directions → |
| This version might be bad and we will not know for hours | Progressive delivery: expose it gradually, with an automatic comparison against the untouched baseline and a defined abort condition.Progressive Delivery: Exposure as a Dial → |
| We need to be able to turn this feature off in seconds, without a deploy | A feature flag decouples release from deployment. It is also new state, new code paths and new debt with an expiry date.Feature Flags: Deploy Is Not Release → |
| The deploy went out and the error rate went up | Stop the rollout first, then decide between rollback, roll forward, or a mitigation like a flag flip. Diagnose after user impact stops.Rollback: Only Useful If It Is Actually Safe → |
| We cannot roll back because the migration already ran | The migration and the deploy are one coupled event. Expand, migrate, contract — and the contract step only after the last version that needs the old shape is gone.Expand, Migrate, Contract → |
| A schema change is going out with this release | Treat the schema change as the risky half. Additive first, dual-write or dual-read across the transition, destructive last and separately.Why Migrations Are the Dangerous Change → |
| The backfill locked the table and requests started timing out | Batch it, throttle it, make it resumable, and run it as its own operation with its own abort switch — not inside a migration step.Backfills → |
| Dropping a column broke the pods that had not been replaced yet | A destructive change is only safe once no running version references the old shape. Contract is a separate deploy, gated on evidence, not a step in the same one.Destructive Migrations → |
| The release is enormous because we batch a quarter of work into it | Change size is the strongest predictor of incident severity you control directly. Smaller, more frequent, individually reversible.Change Size: Why Small Changes Are Safer, and When They Are Not → |
| A branch has been open for six weeks and merging it is terrifying | Long-lived branches convert integration risk into one large event. Trunk-based development with flags moves that risk into small daily increments.Trunk-Based Development → |
| We deployed but the feature is not live for anyone yet | Deployment and release are different events. Knowing which one you are doing determines who is at risk and what rollback means.Deployment Is Not Release → |
| Requests fail during every deploy, in the seconds around shutdown | The process is not handling termination: it must stop accepting new work, drain in-flight requests, and only then exit — and the platform must give it long enough.Graceful Shutdown → |
| A new instance starts getting traffic before it can serve it | Readiness is not liveness. Readiness gates traffic; the load balancer must not add an endpoint until the process says it is able to serve.Probes: Readiness, Liveness and Startup → |
| The pod restarts over and over | Read the previous container logs and the exit code. CrashLoopBackOff is a symptom of a config error, a missing dependency, an OOM kill, or a liveness probe that is failing a healthy process.Reading a Broken Workload → |
| The pod is Pending and never schedules | The scheduler could not place it: no node satisfies the requests, or taints, affinity or topology constraints exclude every node that could.The Scheduler, and Why a Pod Is Pending → |
| The container was killed with exit code 137 | It exceeded its memory limit and the kernel OOM-killed it. Look at real working set versus limit before raising the limit — the leak does not care about the new number.OOMKilled: Over the Memory Limit → |
| Latency is bad but CPU utilisation looks fine | CPU throttling against a quota shows up as latency, not as high utilisation. Check throttled periods, not average CPU.CPU Throttling: The Latency With No Error → |
| Nobody knows what to put for requests and limits | Requests are what the scheduler reserves; limits are what the kernel enforces. Getting them wrong causes both bin-packing failure and throttling, in opposite directions.Requests and Limits → |
| The deploy dropped in-flight requests at the load balancer | Connection draining: deregister first, wait out the health-check and keep-alive window, then terminate. The order is the whole trick.Draining: Stopping Without Dropping → |
| The certificate expired on a Saturday | Renewal must be automated and monitored on remaining validity, not on a calendar reminder. Alert on days-to-expiry, and test the renewal path.Renewal: Automating the Thing That Expires → |
| Clients kept hitting the old address after we changed the record | DNS caching is governed by TTLs you set in advance and by resolvers and clients that may ignore them. Lower the TTL before the change, not during it.DNS in Production → |
| Services find each other by an IP someone wrote down | Service discovery exists because instances are ephemeral. The address must be resolved at call time against a registry the platform keeps current.Service Discovery in Operation → |
| The on-call phone goes off so often that people mute it | Alert fatigue is a systems problem, not a discipline problem. Every alert must be actionable, symptom-based, and tied to something a human can do right now.Alert Fatigue → |
| Alerts fire on CPU but customers complain about errors we never paged on | Page on symptoms users feel — error rate, latency, unavailability — and use cause metrics for diagnosis, not for waking people up.Alert on Symptoms, Not on Causes → |
| The dashboard has ninety graphs and none of them answer "is it healthy" | An operator dashboard answers a small number of questions in a fixed order. Everything else is an investigation tool, not a dashboard.Dashboards an Operator Can Act On → |
| The graph moved but nobody can see what changed at that moment | Deploy annotations. Correlating a change with a signal is the single highest-yield thing you can put on a production dashboard.Deploys on the Same Timeline as the Symptom → |
| The incident has six people in a call and no one is deciding | Someone must own coordination and be explicitly not debugging. The role names are organisation-specific; the separation of coordination from investigation is not.Roles During an Incident → |
| We spent the incident trying to understand the cause while users were down | Mitigate first. Understanding is a post-incident activity; stopping user impact is the incident.Stop the Harm Before You Understand It → |
| The postmortem action item is "be more careful" | That is not an action item. Fix the system that permitted the mistake: a guardrail, a check, a default, a signal — with an owner and a date.Action Items That Change the System → |
| The same incident happened again six weeks later | The learning did not reach the system. Look at whether the action items landed, and whether the write-up found contributing factors or stopped at one convenient cause.Learning Across Incidents → |
| Production is broken and the first question is "what changed" | Start from the change timeline — deploys, config, flags, infrastructure, and dependency releases — before you start from the code.Change Correlation → |
| The runbook says "investigate the issue and resolve it" | A runbook that does not encode understanding is decoration. Concrete commands, decision points, escalation criteria, and what to do if each step fails.Runbook Anti-Patterns → |
| Traffic exceeded what the system can serve | Three separate levers: headroom you provisioned in advance, autoscaling that reacts with lag, and load shedding that protects the system by refusing work deliberately. Autoscaling alone is a tool within capacity management, not an answer to it.Load Shedding → |
| We do not know how much traffic we can actually take | Build a capacity model: what saturates first, at what level, and how much of the fleet can be lost while still serving peak.Building a Capacity Model → |
| The autoscaler scales up and down repeatedly without settling | Oscillation from a lagging or noisy signal. Stabilisation windows, scale-down delays, and a signal that reflects the real constraint rather than a proxy for it.How Autoscaling Fails → |
| The queue backlog grows and CPU-based scaling does nothing about it | Scale on the metric that represents the work waiting — queue depth or age — not on the utilisation of the consumers.Queue-Based Autoscaling → |
| Scaling out exhausted the database connection limit | Connections are a shared budget across every replica, every job and every migration tool. Pool sizes multiply by replica count; the autoscaler does not know that.The Connection Budget → |
| Cache nodes restarted and the database fell over | A cold cache converts a cache-hit workload into an origin workload instantly. Warm on start, stagger restarts, and coalesce concurrent misses.Operating a Cache → |
| Messages are piling up in the dead-letter queue | A DLQ is a queue of unfinished work with no owner unless you give it one: alert on depth and age, and have a documented replay path.Dead Letter Queues Are an Operation → |
| The cron job ran twice, or did not run at all | Schedulers give at-least-once or at-most-once behaviour, never both. Idempotent jobs, overlap policy, and an alert on the absence of a run.Job Scheduler Reliability → |
| A nightly job ran an hour early after a clock change | Local time in production is a bug generator. Run and store in UTC; convert only at the edges where a human reads it.Timezone and DST Failures → |
| The cloud bill jumped and nobody knows why | Attribute cost to drivers — egress, storage class, idle capacity, cross-zone traffic, managed-service tiers — before optimising anything.Cost Drivers → |
| We want to know whether this architecture is affordable at ten times the traffic | Cost per unit of work is the number that scales. Total spend is not comparable across time; cost per request is.Cost Per Request → |
| A service is paging and nobody knows who owns it | Ownership is a prerequisite for operating, not a formality: a named team, an escalation path, and a service that cannot enter production without them.The Ownership Record → |
| We take backups but have never restored one | An untested backup is a hypothesis. A restore drill measures the real recovery time and finds the missing credential, schema or dependency before the disaster does.Restore Drills → |
| We need to know how much data loss and downtime is acceptable | RPO and RTO are business decisions with architectural consequences, and only real if a tested procedure meets them.RTO and RPO → |
| We failed over to the second region and it could not take the load | Failover capacity is capacity you must be paying for or able to acquire immediately. If two regions each serve half of normal traffic, either one must be able to serve all of it.Capacity During Failover → |
| Everyone on the team has permanent production admin | Standing privilege is the risk. Scope by role, grant just in time, expire automatically, and audit what was used.Least Privilege in Production → |
| Someone genuinely needs root on production right now, at 3am | Break-glass access: a real path, deliberately noisy — pre-approved, time-boxed, logged, alerted on, and reviewed afterwards.Break-Glass Access → |
| The same manual operation is done by hand every week | That is toil: manual, repetitive, automatable, and scaling with the system. Measure it, then automate the part you understand.Toil → |
| We automated a procedure and it made the incident worse | Automation scales whatever you encoded, including the mistake. Automate the understood and the reversible; keep judgement human.The Automation Trap → |
| Every team has invented its own pipeline and nothing is consistent | A golden path: an opinionated, supported route that is genuinely easier than rolling your own, not a mandate.Golden Paths → |
| The platform team has become a ticket queue that blocks every change | Gates review each change by hand and become the bottleneck. Guardrails make the unsafe thing impossible or loud and let teams move without asking.Guardrails, Not Gates → |
| Nobody can say what is running in production or who approved it | A release manifest and an audit trail: version, commit, artifact digest, config revision, approver, timestamp — queryable during an incident.The Release Manifest → |
| A new service is about to take real traffic | Run a readiness review against dimensions with evidence: owner, rollback, signals, alerts, runbook, capacity, tested recovery. Status per dimension, never a score.Production Readiness Review → |
| Changing the prompt degraded answer quality and nobody noticed for a week | Prompts, models and tool definitions are deployable production inputs. Version them, evaluate before rollout, and treat a model swap as a deploy.Prompts and Models Are Deployables → |
| We need to stop the model from serving right now | A kill switch that does not require a redeploy, with a defined fallback behaviour that has been exercised.The Agent Kill Switch → |
| Inference spend tripled after a release | Token cost per request is a production metric with a rollout cause: model choice, context growth, retries and tool loops. Attribute it per route before capping anything.Agent Cost in Production → |