Evolutiontelemetryusage attributionburn-downevolution decisions

Consumer-Driven Evolution: Telemetry Before Breakage

"Can we remove this?" is a telemetry query, not a debate. Per-consumer, per-field usage attribution turns evolution decisions from opinions into evidence — and the 12% of mobile users on an old build stop being invisible exactly when you can count them.

Follow the failure

Frame the contract

API design starts with a consumer, a design question and a guarantee — never with a URL.

Design question
Before this contract element changes, do we know — per consumer, with numbers — who depends on it and how much?
Consumers
The provider team is the primary consumer of this lesson's output: every deprecation, removal and version-sunset decision they make is only as good as the attribution behind it. Downstream, every API consumer benefits from breaks that target verified-empty surface.
The promise
With usage attributed per consumer per element, the provider can promise that no announced-safe change breaks a measurable dependency — and that when a break is chosen anyway, everyone it hits was known and contacted first.
RequirementConsumersResource ModelStyleContractValidationAuthorizationErrorsIdempotencyPaginationVersioningObservabilityEvolutionTrade-offs

Evolution decisions are queries, if you built the table

Every evolution decision in this module bottoms out in the same question: *who still depends on the thing we want to change?* Removing Fields Without Removing Consumers needs readers per field; Deprecation as a Process, Not a Label needs a burn-down per surface; Versioning: What a Version Even Promises sunsets need consumers per version; Enum Evolution: The New Value That Broke Old Clients needs to know which SDK vintages will meet the new value. Teams without attribution answer these with grep, memory and hope. Teams with it answer in a dashboard, and the difference shows up as incident count.

The canonical scenario: the team wants to drop a legacy response block that "nobody uses anymore". The query says: web — zero reads for six months; partners — two keys, both migrated last quarter; mobile — 12% of the active fleet is on builds ≤ v4.1, which render this block on the profile screen. Twelve percent of a million monthly users is 120,000 people whose app breaks. The block is not removable — not this quarter — and no amount of engineering opinion in a meeting could have produced that answer. What it *is* is a plan: the number, per build version, becomes the burn-down that tells you which quarter the removal lands.

The query that replaced the meeting
usage: response block `profile.legacy_stats` — last 90 days

  consumer            reads/day   trend    note
  web-app             0           →        migrated 2026-01
  key_partner_a       0           →        migrated (ticket #812)
  key_partner_b       0           →        never used it
  mobile ≥ v4.2       0           →        new profile screen
  mobile ≤ v4.1       ~48,000     ↓ 3%/mo  12% of active fleet

  verdict: NOT removable. Gate: mobile ≤ v4.1 under 2% of fleet
  (projected ~11 months) or forced-upgrade decision by product.

Building attribution: identity × element × time

Attribution is three joins. Identity: every request must carry who is calling — API key, OAuth client id, service identity, and for first-party apps a build/SDK version header — which is an argument for per-consumer credentials long before any analytics need (see API Ownership and the Catalog and API Keys: Identity for Applications). Element: which contract surface the request touched — endpoint and version come free from routing; request fields and enum values sent are visible in the payload; *response-field reads* are the famously hard part, approximated by SDK version, measured exactly only with field selection or instrumented SDKs (see Removing Fields Without Removing Consumers for that ladder). Time: trends matter more than totals — a field at 100 reads/day and falling 20% weekly is a migration finishing itself; the same field flat for a year is a permanent dependency.

Two engineering notes keep this honest. Sample the expensive parts — payload-level attribution on 1% of traffic answers "who sends this field" with plenty of confidence at 1% of the cost, though rare-but-critical consumers (the quarterly job) can hide below sample floors, so keep unsampled counters for identity × endpoint at least. And keep cardinality bounded: attribute to *registered consumers* (hundreds), never to end users (millions) — this is the same discipline as API Metrics: Rate, Errors, Duration, Sizes, applied to evolution data. What you must not do is log payloads wholesale to figure usage out later: that turns an analytics need into a data-retention and secrets liability (see API Logging Without Leaking).

  • Identity: per-consumer credentials + a User-Agent/SDK-version convention your SDKs enforce; unattributable traffic gets its own alarm, because it is exactly the traffic that will surprise you.
  • Endpoint × version: free from routing; the minimum viable attribution, enough for version sunsets and endpoint deprecations.
  • Request fields / enum values sent: sampled payload inspection at the gateway or middleware; answers "who still sends X" directly.
  • Response-field reads: SDK-version proxy → sparse-fieldset or GraphQL exact measurement → instrumented SDK accessors, in rising fidelity and cost.
  • Retention: keep at least one full business cycle — the consumer that calls once a quarter is invisible in a 30-day window.

From numbers to decisions: gates, weights, and the consumers who count

Telemetry informs the decision; it does not make it. Raw counts need weighting: 10,000 reads/day from one partner's retry-happy integration is less important than 10 reads/day from the endpoint your largest customer's billing depends on. The practical form is a gate written into each evolution plan — *removable when: zero reads over one business cycle, or all remaining readers individually contacted and dispositioned* — with disposition meaning migrated, exempted, or consciously accepted as breakage by someone with authority to accept it (see Deprecation as a Process, Not a Label for the endgame options).

Two failure modes deserve names. Threshold gaming: "under 1% of traffic" sounds like a removal gate but 1% of a big API is thousands of daily calls with humans behind them — gates should be zero-or-named, not percentage-fig-leaves. Survivorship blindness: telemetry only sees consumers who are still calling; the integration that broke last month and silently fell back to a manual process is invisible in current traffic but very much a dependency. Attribution is necessary, not sufficient — it tells you who to talk to, and the talking is still part of the process.

Decision by aggregate
1Proposal: remove `legacy_stats`
2Evidence: "usage is down to 0.8% of
3 total API traffic"
4Decision: below 1% thresholdremove
5
6# 0.8% = 48k reads/day
7# = one entire mobile fleet segment
8# threshold was a fig leaf, not a gate
Decision by named burn-down
1Proposal: remove `legacy_stats`
2Gate: zero reads over 1 business cycle,
3 OR every remaining reader named,
4 contacted, and dispositioned.
5
6Status: mobilev4.1 = 48k reads/day
7disposition: wait for fleet decay
8 (tracked monthly), revisit at <2%
9 with product sign-off on the rest.

Percentages hide people. The gate on the right cannot be satisfied by traffic growth elsewhere diluting the number — only by the actual dependency going away or being consciously accepted.

Key points

  • Every evolution decision — removal, sunset, deprecation pacing — bottoms out in "who depends on this", which is a telemetry query if you built attribution.
  • Attribution is identity × element × time: per-consumer credentials, per-surface usage, and trends over at least one full business cycle.
  • Response-field reads are the hard case: approximate with SDK versions, measure exactly with field selection or instrumented SDKs.
  • Sample payload-level attribution for cost, keep unsampled identity × endpoint counters for safety, and bound cardinality to registered consumers.
  • Gates are zero-or-named, never percentages: 0.8% of a big API is a fleet of real users hiding behind a decimal.
  • Telemetry sees only current callers — the consumer who already broke and gave up is invisible; contact remains part of the process.

Follow the failure

How the contract fails or gets misused, hop by hop — and what it costs when it completes.

  1. 1
    Team → infrastructure: shares one API key across all first-party apps and never adds an SDK-version header — traffic is a single undifferentiated stream.
  2. 2
    Team → decision: wants to remove a legacy block; with no attribution, the evidence is "the web app stopped using it and traffic looks flat".
  3. 3
    Team → deploy: ships the removal; total traffic indeed barely moves — 0.8% was the old mobile fleet.
  4. 4
    Users → support: 120k users on old builds hit broken profile screens; app-store reviews arrive before the incident channel does.
  5. 5
    Team → retrofit: adds attribution *after* the incident, discovers three more surfaces with the same invisible dependency shape, and freezes all removals for two quarters.
What breaks
  • Breaks land on the least-visible consumers by construction — the old fleets and unstaffed integrations that aggregate metrics dilute into noise.
  • Without attribution, safe changes are also blocked: unable to prove a surface is unused, the team keeps everything forever, and the contract only grows.
  • Post-incident, evolution stalls org-wide: every removal proposal now requires the evidence nobody can produce, so nothing is ever cleaned up.

Design, observe, evolve

A contract decision is incomplete until you know how you would notice it failing and how it changes later.

Design the contract
  • • Issue per-consumer credentials and enforce an SDK/build-version convention from day one — attribution starts with identity, and identity cannot be retrofitted onto historical traffic.
  • • Instrument at the layer that sees everything (gateway or shared middleware): identity × endpoint × version unsampled, payload-level element usage sampled.
  • • Write a numeric gate into every evolution plan — zero over a business cycle, or a named-and-dispositioned reader list — and make it the removal ticket's exit criterion.
  • • Alert on unattributable traffic and on new consumers adopting deprecated surfaces; both are attribution rot in progress.
Observe in production
  • • Burn-down charts per deprecated element, segmented by consumer, are the standing dashboard; plateaus trigger the next outreach escalation rather than quiet deadline slips.
  • • Track the share of traffic that is fully attributed (identity + version known); below ~99%, every evolution decision inherits the unattributed remainder as unpriced risk.
  • • Watch trend inflections: a dependency that stops decaying often means a new consumer adopted the old surface — find them before the gate assumes decay resumes.
Evolve without breaking
  • • Attribution compounds: each campaign leaves better instrumentation, and evolution decisions get faster and safer as coverage grows — the platform learns which of its promises are load-bearing.
  • • Publish consumer-facing usage of deprecated surfaces ("your key called these 3 sunsetting endpoints last week") and consumers start migrating themselves — the telemetry flows both ways.
  • • The same data answers design questions prospectively: which fields nobody reads is also the input to slimming default responses (see [[over-under-fetching]]).
What it costs
  • • Attribution infrastructure is real cost — credentials, headers, sampling pipelines, dashboards — paid continuously for benefits that arrive only when change is needed.
  • • Payload-level usage data is sensitive by nature; sampling, field-name-only capture and short retention keep it from becoming a shadow copy of customer data (see [[api-logging]]).
  • • Evidence-gated evolution is slower than decisive breakage; for an internal API with three known consumers, a conversation beats a telemetry platform (see [[public-vs-internal-apis]]).

Misconceptions

Claim
“Overall traffic to the endpoint is flat, so the field/feature is unused.”
Reality
Aggregates hide segments. A dependency carried by 12% of a mobile fleet is invisible in total traffic and very visible in app-store reviews. Only per-consumer, per-element attribution can say "unused".
Claim
“Usage telemetry means logging requests — we already have access logs.”
Reality
Access logs give identity × endpoint at best. Evolution decisions need element-level attribution (fields, enum values, versions) with bounded cardinality and deliberate sampling — a designed dataset, not a byproduct.
Claim
“If telemetry says zero, the removal is safe.”
Reality
Zero over a window shorter than the consumer's business cycle misses the quarterly job; zero also misses the consumer who already broke and stopped calling. Zero over a full cycle plus a completed contact list is the actual gate.