Reliability EngineeringoptimizationIntermediate

The bill is 20x the estimate a week after launch

Symptoms

  • Projected LLM spend was ~$400/month; the first week alone cost $2,100.
  • Traffic is only slightly above the estimate — request volume does not explain the gap.
  • Average tokens per request are far higher than in the demo.
demo avg: 3,200 prompt tokens/request, 1 model call
prod avg: 28,500 prompt tokens/request, 4.1 model calls/request
top cost driver: full document corpus (all 40 chunks) injected every request, no cap
model: using the largest model for every call, including classification

Investigate

Inspect areas in any order (0/5 inspected). When you think you know the root cause, commit to it.

Context size per request
Model selection
Caching
Retry policy
Request volume