"What Changed?" — Deploy Markers and the Invisible Deploys
The highest-yield first question in any performance incident is what changed, and it is only answerable in seconds if changes appear on the same time axis as the metrics. Code deploys are the easy part; config pushes, feature flags and someone else's release are the ones that leave no mark.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Change events belong on the metric chart
The overwhelming majority of performance regressions are caused by a change, and the overwhelming majority of those changes are recent. This makes "what changed?" the highest-yield opening question in incident debugging — and its value is entirely determined by how fast it can be answered. Asking in chat takes twenty minutes and produces a partial answer. Reading vertical lines on the same chart as the latency takes five seconds and produces a complete one.
A deploy marker is a vertical annotation on time-series charts carrying the release identifier and a link to the diff. The value is not the annotation itself but the *comparison*: a step change that begins exactly at a marker is a strong lead, and a step change that clearly begins before the nearest marker kills the "it was the deploy" hypothesis in one glance. Both outcomes are worth having, and the second one is worth more, because it is the one that stops a team from investigating an innocent release.
Markers also make the boring case fast. Most incidents where the answer is "nothing shipped" would otherwise consume ten minutes establishing that, and those ten minutes come out of the mitigation window. A dashboard that shows six hours of markers and none in the incident window redirects the investigation outward — to dependencies, traffic and data volume — immediately (Debugging an Incident in Progress).
p99 (ms)
4000 | ╭──────────────
| ╱
3000 | ╱
| ╱
2000 | ╭─╯
| ╭─╯
1000 | ╭───╯
| ────────────────────╯
200 |───
+───┬────────┬──────────┬─────────┬──────────┬────→
13:00 13:30 ▲ 13:40 14:00 ▲ 14:30
│ │
(nothing) deploy v2.4.1
Reading: the rise begins ~13:38, twenty-two minutes BEFORE the
deploy. The deploy is a coincidence. Without markers, the team
would have spent that window reading the v2.4.1 diff.The invisible deploys
Code deploys are the change type everyone instruments, and they are a minority of the changes that break production performance. A feature flag flipped from a web console changes behavior for a percentage of traffic with no build, no pipeline and no artifact. A config push changes a connection pool size or a timeout. A dependency team ships their own release, invisible in your deploy log and fully capable of doubling your latency (Fan-Out: Waiting for the Slowest of Seven). A schema migration completes. A cron job starts. A certificate rotates. A cloud provider migrates your instance.
These are genuinely harder to capture, and the fix is organizational as much as technical: every system that can change production behavior should emit a change event to the same stream that deploys use. That includes the flag service, the config service, the infrastructure pipeline, and — critically — the equivalent streams from the teams you depend on. A shared change feed across services is one of the highest-leverage pieces of observability infrastructure a platform team can build, and it is almost never anybody's priority until an incident makes the case.
Feature flags deserve specific attention because they combine invisibility with gradual rollout. A flag ramping from 1% to 50% over an hour produces a *gradual* latency rise that looks like organic traffic growth, not a step change at a marker. If your flag system does not emit ramp events, this is close to undiagnosable from metrics alone — and gradual regressions are exactly the kind that survive long enough to become "how it has always been".
| Change type | Typical visibility | Shape it produces | How to make it visible |
|---|---|---|---|
| Code deploy | Good — deploy pipeline emits events | Step change at the marker | Annotate every dashboard from the pipeline, with a diff link |
| Feature flag flip | Poor — changed from a console | Step change, often for a traffic subset only | Flag service emits change events, including ramp percentage changes |
| Flag gradual ramp | Very poor | Gradual rise resembling organic traffic growth | Emit an event per ramp step; graph the exposed percentage as a metric |
| Config / env change | Poor to moderate | Step change, often at process restart rather than push time | Config service emits events; record the restart, not just the push |
| Dependency release | None from your side | Step change in one downstream span only | Cross-team change feed, or subscribe to their deploy events |
| Schema migration | Moderate | Step change when the plan flips, which can be much later | Emit migration completion; also watch for delayed plan changes (The Slow Query Workflow) |
| Data volume threshold | None — nothing "changed" | Gradual then sudden, when a plan or a cache tips over | Track table and index sizes as metrics; alert on plan changes |
| Infrastructure / provider | None to poor | Step change confined to specific hosts or a zone | Host- and zone-level dimensions on latency metrics so the subset is visible |
Making the answer take five seconds
The engineering work is small and the payoff is per-incident. Every service dashboard carries markers for its own deploys plus its dependencies' deploys. Flag changes and config pushes emit to the same annotation stream. The change feed is queryable by time range, so "what changed between 13:30 and 14:10 anywhere in the request path" is one query rather than five conversations.
One caution worth stating plainly: markers make correlation extremely easy to see, which makes the correlation-causation error extremely easy to commit. A marker adjacent to a step change is a lead, and the very next step is to establish whether the change could plausibly affect the observed path — read the diff, check whether the flag applies to the affected route, verify the config actually reached the affected hosts (Correlation Is Not the Root Cause). Teams with excellent marker infrastructure and no mechanism discipline roll back innocent deploys with great confidence.
The second-order benefit is that a good change feed makes gradual regressions detectable at all. Plotting "percentage of traffic with flag X enabled" alongside p99 turns an undiagnosable slow drift into an obvious correlation — and drift is where most performance debt accumulates, precisely because nothing pages for it (Regression or Tuesday? Telling a Real Change from Noise).
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| checkout-api deploys | v2.4.1 at 14:00 | After the deviation began at 13:38. Not the cause; stop reading the diff. | normal |
| payment-api deploys (dependency) | v8.2.0 at 13:35 | Three minutes before the deviation, in the span that got slow. This is the lead. | smoking gun |
| Feature flags | `fast_checkout` 5% → 25% at 11:00 | Earlier and gradual; would explain a drift, not a 13:38 step. Note it, do not chase it. | suspect |
| Config pushes | None in window | Rules out pool and timeout changes as an explanation. | normal |
| Scaling actions | Scale-out 14:06 (autoscaler) | A response to the incident, not a cause. Belongs on the timeline as a reaction (Autoscaling Lag: The Gap Where the Outage Lives). | normal |
| Schema migrations | None in 48h | Rules out plan changes from new indexes or column types. | normal |
Key points
- "What changed?" is the highest-yield first question, and its value depends entirely on being answerable in seconds rather than in a chat thread.
- Markers are as valuable for exoneration as for accusation — a deviation starting before the nearest marker kills the default hypothesis instantly.
- Code deploys are a minority of production changes; flags, config, dependency releases, migrations and data growth are the invisible ones.
- Gradual flag ramps produce drift that resembles organic growth — graph exposed percentage as a metric or the regression is undiagnosable.
- Excellent markers plus no mechanism discipline produces confident rollbacks of innocent releases; the marker is a lead, not a verdict.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Change → system: a deploy, flag, config push or dependency release alters behavior on some fraction of traffic.
- 2System → metric: the affected fraction shifts a percentile or a resource curve, gradually if the change is ramped, as a step if it is not.
- 3Metric → responder: the responder sees the deviation but has no change data on the same axis, so ordering must be established by asking humans.
- 4Responder → chat: twenty minutes of "did anyone ship anything?" produces a partial answer covering only code deploys from teams present in the channel.
- 5Partial answer → wrong lead: the most recent visible deploy is investigated regardless of whether it preceded the deviation, while the invisible change stays invisible.
- • "There is a marker right there, so that is the cause" — check whether the deviation actually begins at the marker or before it, then read the diff for a plausible mechanism.
- • "No deploys in the window, so nothing changed" — flags, config, dependency releases, cron jobs and data growth all change production without a deploy marker.
- • "The autoscaler scaled out at 14:06, that is suspicious" — scaling actions during an incident are usually reactions to it; check whether the action preceded or followed the deviation (Autoscaling: Scaling on the Right Signal).
- • "The flag was flipped three hours ago so it cannot be related" — a ramp that crossed a saturation threshold produces a delayed cliff, and a flag enabled long ago can interact with a traffic peak today.
- • "It is a step change, so it is a deploy" — a dependency's deploy, a cache node dying, or a plan flip produce identically shaped step changes.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • A queryable change feed spanning code deploys, flag changes, config pushes, scaling actions and migrations, with timestamps and identifiers.
- • Dependency deploy events for every service in your request path, not just your own — the span that got slow is usually not the one you deployed.
- • Feature-flag exposure as a time series (percentage of traffic enabled), so ramps become visible as a curve rather than an invisible drift.
- • Process restart times distinct from config push times, since a config change often takes effect only at restart, sometimes hours later.
- • Table, index and dataset size metrics, which are the "change" behind regressions where nothing was deployed at all.
- • Emit change events from every system that can alter production behavior — pipeline, flag service, config service, infrastructure automation — into one annotation stream.
- • Overlay that stream on every performance dashboard automatically, including dependency deploys for services in your request path.
- • Graph feature-flag exposure percentage as a first-class metric so ramps are visible as curves.
- • Record process restart times as separate events from config pushes, since the delay between them is where "nothing changed" incidents hide.
- • Establish a cross-team change feed so a dependency's release is visible from your dashboard without asking anyone.
- • During the next incident, "what changed in this window?" is answered from one query in under a minute, with no chat thread.
- • A deliberate canary or ramp shows up on the dashboard as an annotated, attributable change.
- • Reverting the identified change returns the metric to baseline — the actual confirmation that the marker pointed at a cause and not a coincidence.
- • Post-incident, the reconstructed timeline contains no "we think something changed around then" entries ([[incident-timeline]]).
- • Annotating every dashboard with every change type produces visual noise; filter by relevance to the service, or the markers become wallpaper.
- • A cross-team change feed requires organizational agreement, which is slower to obtain than any of the technical work.
- • Fine-grained flag events add write volume to the annotation store and can themselves become a cardinality problem ([[cardinality]]).
- • Markers make it very easy to blame the nearest deploy — the tooling actively encourages the correlation-causation error unless mechanism discipline is enforced.
- • Treat a missing change source as an action item whenever an incident review contains "we could not tell what changed".
- • Alert on unexpected change events in frozen windows (peak traffic periods, code freezes) where a change is itself the anomaly.
- • Include the change feed in the incident review template so its gaps are visible every time rather than only when they hurt.
- • Automatically annotate CI benchmark results with the same change stream so regressions are attributable before they reach production (Regression or Tuesday? Telling a Real Change from Noise).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe chart, change feed and timings are constructed to show the reasoning. Which change sources exist and how well they are instrumented varies enormously by organization.
- ENVIRONMENT-SPECIFICThe feasibility of a unified change feed depends on your deployment, flag and config tooling. Some stacks emit change events natively; others require a shim per source.