Every Optimization Buys Something and Sells Something
Caching buys database load and sells freshness. Compression buys bandwidth and sells CPU. Batching buys throughput and sells latency. There is no move that is purely faster — and the ones that appear to be are usually selling reliability quietly.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The standard trades, stated as trades
Most performance techniques are conversions: they turn one resource into another, or one property into another. Caching converts freshness into speed. Compression converts CPU into bandwidth. Replication converts consistency into read capacity. Batching converts latency into throughput. Prefetching converts wasted work into reduced waiting. None of these is free, and stating the conversion explicitly is what turns a plausible-sounding suggestion into an engineering decision.
The failure mode is not choosing wrongly — it is not noticing there was a choice. "Add a cache" sounds like a pure improvement, so the invalidation strategy, the staleness window, the stampede behavior and the memory cost get decided by default rather than by design (Cache Stampede: Everyone Misses at Once). Six weeks later, a stale price is shown to a customer, and the retrospective calls it a bug rather than the cost that was accepted implicitly.
Write the price into the change description. "Adds a 60s cache on the pricing endpoint; customers may see a price up to 60 seconds stale; invalidated on write; stampede-protected by single-flight; adds ~2GB memory per node." That sentence takes a minute to write and makes the decision reviewable by someone who knows the business constraint you do not.
| Technique | Buys | Sells | Fails when |
|---|---|---|---|
| Caching | Lower backend load, lower p50 | Freshness, invalidation complexity, memory | Data must be current, or invalidation is wrong and nobody notices for weeks (A 95% Hit Rate Tells You Almost Nothing) |
| Compression | Bandwidth, transfer time on slow links | CPU on both ends, small payload overhead | Payloads are small or already compressed; CPU is the constraint (CPU Saturation: When Cores Become the Queue) |
| Read replicas | Read capacity, isolation from writes | Replication lag, cost, read-after-write surprises | The app reads its own writes immediately (Replication Lag: Reads That Are Correct and Stale) |
| Batching | Throughput, fewer round trips, less per-item overhead | Latency for the first item, partial-failure complexity | Requests are latency-sensitive, or a batch failure loses everything (Batch APIs and Partial Failure) |
| Bigger connection pool | Concurrency, less pool wait | Database memory and contention; moves the queue downstream | The database is the constraint — you queued in a worse place (Connection Pool Saturation: Waiting in Front of an Idle Database) |
| Prefetch / speculative work | Perceived latency | Wasted capacity on unused work; amplification under load | Load is high — speculation competes with real work exactly when it hurts most |
| Denormalization | Read speed, fewer joins | Write amplification, consistency risk, storage | Write-heavy workloads, or the copies drift apart |
| More instances | Capacity | Cost, per-instance overhead, shared-dependency pressure | The constraint is a shared resource that scaling multiplies pressure on (The Bottleneck Moves After Every Fix) |
Performance versus reliability: the quiet sales
Some optimizations sell reliability, and they are dangerous because the price only appears during failures — which is to say, precisely when you cannot afford it. Shortening a timeout improves p99 by cutting off slow requests, and turns transient slowness into errors (Timeouts: The Latency Contract Nobody Writes Down). Removing retries reduces load amplification, and turns recoverable blips into user-visible failures. Disabling fsync makes writes dramatically faster and makes crash-durability a maybe. Enlarging a cache improves hit rate and lengthens the window where stale data is served during an invalidation bug.
None of these is wrong. Some are excellent trades: a cache that can serve stale data during a backend outage is a reliability *win*, not a loss, and a shorter timeout with a fallback is often better than a long wait. The failure is making the trade without naming it, so nobody with the authority to accept the risk was asked, and nobody wrote the alert that would show the price being paid.
A useful test: for each optimization, describe what happens to a *failing* system, not just a healthy one. Batching under a healthy backend is efficient; batching when the backend is failing means one failure loses a hundred items. A larger pool under normal load reduces waiting; under a database slowdown it delivers more concurrent load into an already-struggling database (Queueing: Why Systems Get Slow Before They Get Broken). Performance changes reshape failure behavior, and the failure behavior is the part nobody tests.
1PR: "Reduce checkout p99"2 - client timeout: 30s → 2s3 - retries: 3 → 04 - fsync on order write: on → off5 6result (healthy): p99 2.4s → 0.6s ✓ shipped, celebrated7 8result (degraded backend, six weeks later):9 · 2s timeout fires on requests that would have completed at 2.4s10 · no retry → transient blips become customer-visible failures11 · error rate 0.1% → 9%12 · a crash loses the last ~200ms of committed orders13 14review: "unexpected fragility during payment provider slowdown"1PR: "Reduce checkout p99 — trades stated"2 - client timeout 30s → 4s3 SELLS: requests in the 4–30s range now fail instead of succeeding4 GUARD: p99 of the dependency is 1.9s; 4s is >2x headroom5 ALERT: timeout rate > 1% pages6 - retries 3 → 2, with exponential backoff + jitter7 SELLS: slightly less resilience to single blips8 BUYS: removes the retry-storm amplification we saw in INC-2149 - fsync: UNCHANGED10 rejected: durability is not ours to trade for 40ms11 12result: p99 2.4s → 0.9s, error rate flat, durability intact.The second version is not more cautious — it is more specific. Each trade names what is being sold, checks it against a measured value, and adds the alert that shows the price being paid. The fsync change was rejected outright because durability was not the author's to trade, which is exactly the kind of decision that needs to be visible in a diff rather than buried in a config.
Performance versus cost, and the direction people assume
Faster is frequently more expensive, and the assumption runs the other way often enough to cause real budget surprises. More replicas, more RAM for a larger cache, premium storage tiers, cross-region deployment to reduce latency, keeping instances warm to avoid cold starts — all buy latency with money, continuously, forever (Cost per Request: The Other Performance Metric).
The opposite also happens and is worth exploiting: efficiency work reduces both latency and cost simultaneously. Removing an N+1 makes requests faster *and* reduces database load, which defers a scaling purchase (The Comb: N+1 as a Visible Shape). Fixing a memory leak improves tail latency *and* lets you run smaller instances. These are the highest-value optimizations precisely because they refuse the trade — and they are what "efficiency" means as distinct from "capacity" (Capacity or Efficiency: Which Problem Are You Solving?).
When you must buy latency with money, express it as a rate: "reducing p99 by 120ms costs approximately $4,200/month in additional replicas". That framing lets a product owner decide whether 120ms is worth it, which is their decision and not yours. Presenting the same change as "we made checkout faster" hides the recurring cost inside an engineering win.
| Change | Latency effect | Cost effect | Notes |
|---|---|---|---|
| Remove an N+1 query | Large improvement | Reduces DB load — cost down | Refuses the trade; do these first (The Comb: N+1 as a Visible Shape) |
| Fix a memory leak | Improves tail (fewer GC/OOM events) | Allows smaller instances — cost down | Refuses the trade |
| Add read replicas | Improves read latency under load | Linear cost increase, forever | Buys capacity with money; also adds lag (Replication Lag: Reads That Are Correct and Stale) |
| Larger cache tier | Improves hit rate and p50 | Memory cost; possibly a bigger instance class | Diminishing returns past the working-set size |
| Multi-region deployment | Large improvement for distant users | Substantial: infra, data sync, operational complexity | The only fix for propagation delay (Cross-Region Latency Is Physics, Not Configuration) |
| Keep instances warm | Removes cold-start tail | Pay for idle capacity | Common in serverless; the trade is explicit and continuous |
| Compress responses | Improves transfer on slow links | CPU cost up, bandwidth cost down | Net direction depends on which you pay more for |
Key points
- Most performance techniques are conversions — freshness into speed, CPU into bandwidth, consistency into read capacity, latency into throughput.
- The failure is not choosing badly but not noticing there was a choice; unstated trades get decided by default and discovered in an incident.
- Optimizations that sell reliability are the dangerous ones, because the price is only paid during failures.
- Describe what each change does to a *failing* system, not just a healthy one — performance changes reshape failure behavior.
- Efficiency work that reduces latency and cost together refuses the trade entirely; do those before anything that buys latency with money.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Optimization → resource: a change converts one resource or property into another, improving the targeted metric as designed.
- 2Change → unmeasured axis: the paired cost lands on an axis nobody instrumented — staleness, CPU headroom, durability, first-item latency.
- 3Healthy load → no signal: under normal conditions the cost is invisible, so the change validates cleanly and ships with confidence.
- 4Degraded conditions → price due: a dependency slows, a node crashes, an invalidation is missed, and the sold property is now the failure mode.
- 5Incident → misattribution: the review blames the triggering condition rather than the trade, so the same trade is made again elsewhere.
- • "This change is purely an improvement" — then you have not found the axis it sells on. Caching sells freshness, batching sells latency, shorter timeouts sell success rate.
- • "Faster is cheaper" — sometimes, and only for efficiency work. Capacity-based latency improvements cost money continuously.
- • "The load test passed" — load tests almost always run against a healthy backend, which is the exact condition under which the sold property is invisible.
- • "We can tune the trade later" — staleness windows and durability settings become load-bearing quickly, and reverting them after downstream systems depend on the behavior is a migration, not a config change.
- • "Turning off fsync is fine, we have replicas" — replicas propagate what was committed; unflushed writes are not committed. This trade is frequently made without anyone realizing it was made (Disk and Storage: Latency, Throughput, IOPS and the fsync Tax).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • The paired cost metric for every optimization: staleness for caching, CPU for compression, replication lag for replicas, first-item latency for batching.
- • Behavior under a degraded dependency, not just under healthy load — load tests that include a slow or failing backend ([[load-testing]]).
- • Cost per request before and after, so latency bought with money is visible as a rate rather than a one-time engineering win ([[cost-per-request]]).
- • Error rate and timeout rate alongside latency, since the fastest way to improve p99 is to fail slow requests sooner.
- • Cache staleness distribution (age of served data), which is the metric almost nobody has and every cache incident needs.
- • Write the trade into the change description: what it buys, what it sells, and the measured value that makes the price acceptable.
- • Add the alert for the sold property at the same time as the change — staleness age, timeout rate, CPU headroom, replication lag.
- • Prefer efficiency work (N+1 removal, leak fixes, algorithmic improvements) that improves latency and cost together, before anything that trades.
- • Test the change under degraded conditions — slow dependency, failing backend, cold cache — not only under healthy load.
- • Escalate trades that sell properties you do not own: durability, correctness and data freshness usually belong to someone else's risk budget.
- • The targeted metric improves by the predicted amount, and the paired cost metric moves by no more than the amount you accepted.
- • A degraded-dependency test shows the failure behavior you predicted rather than a new one.
- • Cost per request moves in the direction and magnitude you stated, measured over a full traffic cycle rather than a quiet hour.
- • The alert on the sold property stays quiet under normal operation — if it fires immediately, the price was higher than estimated.
- • Stating every trade explicitly slows down small changes and can turn a one-line config edit into a design discussion.
- • Alerting on every sold property adds alerts to a rotation that is probably already noisy ([[alert-fatigue]]).
- • Degraded-dependency testing requires fault injection infrastructure that takes real effort to build and maintain.
- • Refusing all trades leaves performance on the table; the goal is deliberate trades, not conservatism.
- • Alert on the sold property permanently, not just during rollout; these costs drift as traffic and data change.
- • Record the trade in an architecture decision note so the next engineer knows the staleness window was chosen rather than inherited.
- • Include degraded-dependency scenarios in the regular load-test suite so failure behavior is re-verified as the system changes.
- • Review recurring costs periodically — latency bought with money keeps costing money long after anyone remembers why (Cost per Request: The Other Performance Metric).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe PR examples, latencies and dollar figures are constructed. Which direction a given trade runs depends on your cost structure — bandwidth-expensive and CPU-expensive environments make opposite decisions about compression.
- WORKLOAD-SPECIFICWhether a trade is acceptable is a business question, not a technical one. A 60-second staleness window is fine for a product catalogue and unacceptable for an order book.