CapacityCLOUD-SPECIFICGENERAL

Cost Drivers

What infrastructure spend is actually made of, expressed as what each component scales with rather than what it costs.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

When the bill grows, what is growing — and what is it growing with?

The problem

Bills are organised by product line, which is how the provider sells, not by driver, which is how you would change it.

What teams do first

Compute is the bill. Look at the instance count and the instance size, and everything else is rounding.

How it breaks

Storage is cumulative. Compute is a rate you can stop paying; stored bytes keep costing until someone deletes them, and retention defaults mean nobody has ever decided to.

How it breaks in production

Storage is cumulative. Compute is a rate you can stop paying; stored bytes keep costing until someone deletes them, and retention defaults mean nobody has ever decided to.
Data transfer is metered independently of the compute that generated it, and chatty service-to-service patterns across zones or regions generate a lot of it without appearing in any application metric.
Observability data grows with cardinality, not with traffic. One new high-cardinality label can multiply a metrics bill without a single extra request (Cardinality: The Label That Took Down Monitoring in Performance).
Non-production is often a large share, because it is sized like production and runs on nights and weekends when nobody uses it (Idle Capacity).
Per-request managed services move cost from a fixed fleet into a variable that rises exactly with success, which is fine until it is the dominant term and nobody modelled it.
Compute itself has two distinct terms: what you provisioned and what you used. Only the first is billed, and the gap between them is invisible on any application dashboard.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

Every cost component is a quantity times a duration or count. Knowing which of those two moves tells you which lever exists: fewer things, smaller things, or for less time.
The components fall into three behaviours. Rate-like costs stop when you stop (compute hours, provisioned throughput). Accumulating costs persist until removed (stored bytes, retained logs, snapshots). Event-like costs scale with work (per-request services, transfers, API calls).
Accumulating costs are the ones that surprise, because they grow with no change to the system. A steady write rate is a steadily rising storage bill by construction.
The scaling variable is what matters for prediction. Compute scales with traffic if you scale, and with time if you do not. Storage scales with write rate times retention. Transfer scales with the topology of your call graph, not with request count alone.
Waste and cost are different questions. A large bill for work that produces value is not waste; a small bill for something nobody uses is. The drivers tell you the first, ownership and usage tell you the second.
Cost drivers interact with reliability decisions in both directions: replication multiplies storage, redundancy multiplies compute, and cross-domain redundancy multiplies transfer (Headroom).

The drivers, and what each one scales with

Read the second column as the forecast and the third as the lever. Anything in the accumulating class is on a trajectory unless a policy stops it.

Driver	Scales with	Behaviour	The lever that exists
Provisioned compute	Instance size times hours running	Rate	Fewer, smaller, or off when idle
Provisioned-but-unused compute	The gap between requests and actual usage	Rate	Right-sizing against measured usage (Overprovisioning)
Object and block storage	Write rate times retention, times replicas	Accumulating	Lifecycle expiry and tiering, applied at creation
Backups and snapshots	Frequency times retention times size	Accumulating	A retention policy tied to the recovery objective (RTO and RPO)
Data transfer	Bytes crossing a zone, region or internet boundary	Event	Change the call topology, not the volume of work
Per-request managed services	Successful and failed requests alike	Event	Fewer calls per request; caching; batching
Logs	Volume times retention	Accumulating	Level discipline and sampling at the source
Metrics	Number of distinct time series, not request count	Accumulating	Cardinality limits on labels (Cardinality: The Label That Took Down Monitoring in Performance)
Traces	Sampled spans times retention	Accumulating	Sampling strategy that keeps the interesting traces
Non-production	Number of environments times hours running	Rate	Schedules, and ephemeral rather than permanent (Ephemeral Environments)
Idle and orphaned resources	Nothing — they scale with neglect	Rate	Ownership and detection (Idle Capacity)

The three behaviours

Grouping drivers by behaviour rather than by product tells you when to expect a surprise. Rate-like costs are stable and boring. Event-like costs move with the business, which is usually acceptable. Accumulating costs rise while the system does nothing new, which is why they are the ones that appear as a mystery.

How each behaviour moves when nothing changes

1rate-like        cost(t) = quantity x price-per-hour
2                 flat while the fleet is flat
3                 lever: fewer, smaller, or stopped
4 
5event-like       cost(t) = events(t) x price-per-event
6                 tracks traffic, including failed traffic
7                 lever: fewer events per unit of work
8 
9accumulating     cost(t) = cost(t-1) + inflow - expiry
10                 rises with a constant inflow and no expiry
11                 lever: expiry policy, or the inflow itself

The third line is the one worth internalising: with a steady write rate and no expiry, the bill rises forever with no change to the system. That is not an anomaly to investigate; it is the definition of the term.

When cost work removes something you needed

GENERALThese trade-offs exist on any platform. What differs is the granularity of the controls: some managed observability products let you set per-signal retention and sampling, while a self-hosted stack makes retention a storage decision you implement yourself.

Every row here is a saving that worked. The problem is what else it did. These are the cases where a cost review should require the same care as a production change, because that is what it is.

Trigger	Symptom	Cause	Response
Log retention shortened	An incident investigation cannot reach the relevant window	Retention chosen from a bill rather than from an investigation requirement	Set retention from the longest realistic investigation, then reduce volume instead (Using Observability, Not Building It)
Uniform trace sampling introduced	Rare failures have no traces at all	Uniform sampling keeps the common case and discards the interesting one	Sample errors and slow requests at a higher rate than successes
Lifecycle expiry applied broadly	Data required for audit or reconstruction is gone	A bucket treated as a cache turned out to be a system of record	Classify data before applying policy; require an owner to approve expiry
Non-production shut down overnight	CI fails at night; a shared dependency is unavailable	Shared services living in an environment labelled non-production	Separate shared platform services from per-team environments before scheduling
Services co-located to reduce transfer cost	A zone failure now takes the whole path down	The transfer existed because the redundancy did	Treat topology changes as availability changes (Capacity During Failover)
Snapshot frequency reduced	Recovery point is further back than the objective allows	Backup cost optimised without reference to the recovery objective	Derive backup frequency from the objective, not from the bill (Backup Operations)

How to do it properly

Most important first.

Break spend down by driver rather than by provider product, and do it per service so it is attributable (Cost Awareness).
For each driver, write what it scales with. That single sentence is what makes forecasting possible and makes an anomaly interpretable.
Attack accumulating drivers with lifecycle policy rather than with cleanup projects: retention rules, tiering and expiry applied at creation (Storage Lifecycle: Hot, Warm, Archive, Delete in Cloud).
Measure the provisioned-versus-used gap for compute and treat it as its own driver, because it is the one nobody sees (Overprovisioning).
Control observability cost at the source — sampling, cardinality limits, log level discipline — not by shortening retention until incidents become undebuggable (The Log Bill and What It Is Buying in Performance).
Put non-production on a schedule. Environments that sleep are the least controversial saving available (Ephemeral Environments).
Look at the call graph for cross-domain chattiness before adding capacity to the services it makes slow.

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone

One testEveryone

What contains it

Contained only by applying lifecycle and sampling changes to one dataset or service at a time and verifying the effect before extending — the failures here are delayed and destructive.

What can go wrong

Failure modes, including of the mitigation

A retention policy shortened to control cost, discovered during the next incident when the relevant logs no longer exist.
Sampling applied uniformly, so the rare requests that actually needed tracing are the ones sampled away.
Lifecycle rules applied to a bucket that turns out to be a system of record, deleting data with a compliance obligation attached.
Non-production shutdown schedules that also stop something production depends on — a shared registry, a test fixture service, a CI runner pool.
Cross-zone traffic reduced by co-locating everything in one zone, which removes the redundancy the topology existed for (Capacity During Failover).
The mitigation failing: a driver breakdown built once, by hand, from a console export, and never updated.

Misreads this invites

"Compute is the bill." Compute is the most visible component. Storage, transfer and observability are frequently large and are the ones that grow without a decision.
"Storage is cheap." Per unit it is small; multiplied by retention, replication and snapshots, and never deleted, it becomes an accumulating term that outgrows the thing it supports.
"We reduced logging, so observability is under control." Cardinality, not volume, usually drives metrics cost, and the two are reduced by different actions.
"Non-production does not matter." It is frequently a large share, and it is the only part of the estate where a mistake in cost work cannot cause a customer-facing outage — which makes it the right place to start.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

A spend breakdown by driver, per service, that a team can read without a finance glossary.
A stated scaling variable for each major driver, and a forecast that has been compared against what actually happened.
Storage growth rate plotted alongside retention policy, so the trajectory is visible before it is a problem.
The provisioned-versus-used ratio for the fleet, tracked over time.

How you get back

Lifecycle and retention changes are the dangerous ones because they are destructive and delayed: the effect appears when the first objects expire, long after the change was reviewed. Apply to one bucket or dataset first and verify what expired.
Sampling and log level changes are configuration and revert quickly, but the data not collected during the change is gone permanently — that gap is not recoverable (Partial and Logical Data Recovery).
Schedule-based shutdowns should be revertible by a single switch, because the first time one takes down something unexpected you want it off immediately.

What to automate, and what stays human

Automate lifecycle: expiry and tiering applied at resource creation by policy, so nothing depends on a future cleanup (Policy as Code).
Automate the driver breakdown from billing and telemetry data so it is current rather than a snapshot.
Automate non-production schedules, with an override a human can use without a ticket.
Keep deletion of anything that might be a system of record human, with a named approver. Automated deletion is the one automation in this lesson that can cause an unrecoverable loss.

What this costs

Cutting observability cost reduces the evidence available during incidents, and the saving is certain while the loss appears only on your worst day (Using Observability, Not Building It).
Aggressive lifecycle rules trade storage cost against the ability to answer questions about the past, including questions you have not thought of yet.
Reducing cross-domain traffic usually means reducing cross-domain redundancy, which is a reliability trade wearing a cost-optimisation label.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

CLOUD-SPECIFICThe driver list reflects metered cloud billing, where transfer, requests and storage are priced separately from compute. On owned hardware most of these are sunk into the capital cost and the equivalent drivers are rack space, power and the capacity you cannot get until the next purchase cycle.
GENERALThe rate / accumulating / event classification transfers everywhere, including to self-hosted estates and to agent workloads where the accumulating term is stored context and the event term is per-call inference.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Cloudegress-costs storage-lifecycle cost-per-service object-storage idle-capacity

Observability & Performancelog-cost-and-sampling cardinality trace-sampling

Domains that do not exist yet

— Testing & Reliability Engineering — how much evidence a system has to retain to remain debuggable, which is the constraint every observability saving runs into.