Why Is My API Slow?
The decision flow from "slow" to a named bottleneck: split the time first, then follow the branch the evidence selects.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Given only "the API is slow", what sequence of checks leads to the actual bottleneck instead of a plausible story?
A dashboard says p95 on /orders is bad. Nobody knows whether that is the database, our code, a dependency or load. The answer needs to be defensible enough to justify the work that follows.
Look at average response time, pick the endpoint at the top, and start optimising the code inside it. Loops first, then add a cache.
Averages hide the problem: a p50 of 40 ms with a p99 of 3 s averages to something that looks fine and describes nobody's experience.
- Averages hide the problem: a p50 of 40 ms with a p99 of 3 s averages to something that looks fine and describes nobody's experience.
- The code inside the handler is frequently a small fraction of the time. Optimising it is real work with no measurable effect.
- Without splitting the time, "slow" and "waiting" are indistinguishable — and they have opposite fixes: less work versus more capacity or shorter holds.
- Caching a result whose cost was contention, not computation, adds an invalidation problem and leaves the contention in place (When Not to Cache).
What is actually happening
- End-to-end latency decomposes into: queueing before your process, runtime scheduling delay, your CPU, waiting on I/O you issued, and response serialization and transfer. Every optimisation targets exactly one of these.
- Utilisation and latency are not linear. As a finite resource approaches saturation, waiting time rises steeply — which is why a service can look fine at 70% and fall over at 85% (Resource Limits).
- Tail latency is a different phenomenon from median latency. The tail is usually queueing, retries, GC pauses, cold caches or one slow shard; the median is usually the work itself.
- A trace answers "where", a profile answers "what", and a saturation metric answers "why now". Using the wrong one wastes the investigation.
- Fan-out multiplies: an endpoint that makes ten downstream calls has a p99 dominated by the slowest of the ten, not by their average (Calling Something You Do Not Control).
The first fork: working or waiting
Everything downstream of this question is different. A process that is working is limited by CPU: profiles are useful, more instances help, caching helps, and doing less work helps. A process that is waiting is limited by something else: profiles show idle stacks, more instances may worsen it, and the fix is shorter holds, better concurrency or a faster dependency.
The decision below is the flow from §89 in its entirety. Each option names the signal that selects it and what the corresponding fix actually costs — because the cost is the part that gets skipped when someone announces the bottleneck in a meeting.
Where is the time going?
when Container CPU near limit or one core pinned; profile shows deep stacks in your own code; latency scales with payload or item count.
cost Profiling in production, then real algorithmic or serialization work. Often the least glamorous and most durable fix (What Serialization Costs).
when Query spans dominate the trace; slow-query log populated; database CPU or IO elevated.
cost Reading plans, adding indexes (write cost, storage, migration risk), or restructuring queries (Schema Migrations from the Application Side).
when Query count per request in the dozens or hundreds; each span short; database busy but not slow.
cost Batching or eager loading, which trades a clean object model for explicit fetch strategy (The N+1 Query Problem).
when One external span dominates; their status page or their own metrics agree; errors and timeouts appear together.
cost Timeouts, fallbacks and a breaker — which means deciding what a degraded answer looks like (Circuit Breakers).
when Every endpoint on the process slows together; loop lag or thread-pool queue depth rises; host CPU may look modest.
cost Moving CPU-heavy synchronous work to a worker, adding a boundary and a serialization hop (Blocking the Event Loop).
when Async side effects are late; consumer utilisation at ceiling; queue age climbing while enqueue rate is flat.
cost More consumers (and more database load), or less work per message (Worker Scaling).
when Handler time high, query time flat, pool waiters sustained above zero.
cost Shorter connection holds — usually restructuring transactions — before any change to pool size (Connection Pool Exhaustion).
when Time concentrated after the last I/O span; latency tracks response size; CPU rises with payload growth.
cost Pagination, field selection or a leaner format — all of which are contract changes (Pagination That Survives a Large Table).
Each branch has one tool that settles it
A branch is only useful if it terminates in a specific check that produces a yes or no. Vague branches ("look at the database") are how investigations stall. The table pairs each symptom with the one artefact that confirms or kills it.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Latency scales with number of items in the response | Small responses fast, large ones slow, CPU-shaped | Serialization and allocation cost proportional to payload | Measure response size percentiles; profile the encode path; paginate |
| One statement appears once per row in the trace | Hundreds of short query spans per request | Lazy loading inside a loop (The N+1 Query Problem) | Count queries per request as a metric; assert on it in tests |
| Query got slow with no code change | Same SQL, much longer duration, starting at a point in time | Plan change after data growth or statistics refresh | Read the plan now and compare with the expected access path |
| Handler slow, dependency fast | Total time far exceeds the sum of spans | Time spent waiting to run: pool acquisition, loop lag, thread queue | Instrument acquire time explicitly; it is invisible otherwise |
| All endpoints degrade in lockstep | Even a static health endpoint gets slow | Process-level: blocked loop, GC pressure, CPU throttling, memory pressure | Check the runtime saturation signal, then the container CPU throttle counter |
| Slow only for one tenant | p99 driven by a small share of requests | Data shape: one account with far more rows, no index on the filtered column | Break latency down by tenant; look at that tenant's row counts (Multi-Tenancy) |
| Slow only right after deploy, then fine | Latency spike that decays over minutes | Cold caches, cold JIT, empty pools — warmup, not regression | Compare against the previous deploy's curve before declaring a regression |
The average is the enemy
The most common reason a slow API stays slow is that the metric being watched cannot see the problem. Averages absorb tails, per-service aggregates absorb per-endpoint effects, and handler-side timers absorb everything that happened before the handler started.
Fixing the measurement is usually a smaller change than fixing the service, and it must come first — otherwise the improvement cannot be demonstrated even if it happens.
const t0 = Date.now()
await handler(req, res)
metrics.avg('orders.ms', Date.now() - t0)
// reports: 118 ms average, steady
// invisible: 3 s p99, all of it spent
// waiting for a pooled connection
// before handler() was even calledconst acquire = timer('orders.pool_acquire_ms')
const conn = await pool.acquire()
acquire.stop()
const q = timer('orders.db_ms')
const rows = await conn.query(sql, args)
q.stop()
const ser = timer('orders.serialize_ms')
const body = encode(rows)
ser.stop()
// histograms, so p50/p95/p99 are answerable
// and the phase that grew is named, not guessedThe second version answers the question the incident actually asks — which phase grew — and it does so from a dashboard instead of from a rebuild. The first version cannot distinguish "we are doing more work" from "we are waiting longer", and those have opposite fixes.
How to build it
Most important first.
- Look at percentiles, per endpoint. p50, p95, p99. Decide which one you are fixing before you start; they usually have different causes.
- Split the time at the first fork: is the process working or waiting? CPU-bound and wait-bound investigations diverge immediately, and the signals that distinguish them are cheap.
- Then split the wait: database, external call, queue, lock, pool acquisition. A trace with a span per call does this in one screenshot (Tracing From the Backend's Side).
- Follow the selected branch to its own tool: a CPU profile for app CPU, a query plan for database time, a client-side span for a dependency, loop lag or thread dumps for scheduling, queue age for consumers, pool wait for contention, payload size for serialization.
- Verify the fix in the same metric you diagnosed with. If the diagnosis was pool wait time, the proof is pool wait time falling — not a subjective "feels faster".
What can go wrong
- Optimising a path that is not on the critical path, so the total does not move.
- Profiling in a development environment where data volumes, concurrency and network latency are all wrong, producing a confident answer about the wrong system.
- Confusing "the database is slow" with "we are asking the database too many times". Same symptom, opposite fixes (The N+1 Query Problem).
- Measuring latency only inside the handler, which by construction cannot show queueing that happened before it started.
- Coordinated omission in the load test: the generator waits for slow responses, so the very requests that were slow never get sent and the tail disappears from the report.
- Lock contention appears as latency only under concurrency, so it reproduces in production and not in a single-user test (Backend Races).
- Cache stampedes are a race: many requests miss the same key at once and all recompute it (Request Coalescing).
- Response time is an oracle. Endpoints whose duration differs between "user not found" and "wrong password" leak account existence; compare secrets in constant time (Credentials and Password Handling).
- Performance dashboards that break down by URL frequently leak identifiers into metric labels. Path templates, not raw paths (The Metrics a Backend Must Emit).
- Slow endpoints are denial-of-service surface: if one unauthenticated request can cost seconds of CPU, cost is the vulnerability (Rate Limiting).
- "Our average response time is 120 ms, so we are fine." Averages are a statement about nobody. Percentiles are about people.
- "The database is the bottleneck" said after seeing high database CPU — high database CPU is equally consistent with your service sending far too many cheap queries.
- "Async will make it faster." Asynchrony improves concurrency under I/O wait; it does nothing for CPU-bound work, and on a single-threaded runtime it can make CPU-bound work worse by hiding it (Backend Runtime Models).
- "Adding instances fixes latency." It fixes queueing-for-your-process. It worsens contention for anything shared behind your process.
- "The endpoint is slow because the code is complex." Complexity and latency correlate weakly; waiting and latency correlate almost perfectly.
Operating it
- Per-endpoint p50/p95/p99 and request rate on one chart — a latency change with a traffic change is a capacity story, without one it is a change story.
- A trace waterfall for a slow request, showing serial calls that could be concurrent and calls that should not be there at all.
- Database time as a share of total request time, per endpoint. The single most decisive number in this investigation.
- Runtime saturation for your stack: event-loop lag, thread-pool queue depth, GC pause time, or worker busy ratio.
- Response payload size percentiles. Serialization cost tracks size, and size grows quietly as fields are added (What Serialization Costs).
- At 10x traffic, the bottleneck usually moves rather than worsening: fix the database and the pool becomes the limit; fix the pool and CPU or the event loop does (The Bottleneck Moves After Every Fix).
- Fan-out endpoints degrade first at scale, because the probability that at least one downstream call is slow rises with the number of calls.
- At high request rates, per-request logging and full-fidelity tracing become bottlenecks of their own; sample and keep the slow tail.
- Tracing everything costs money and storage; sampling risks missing the rare pathological request. Sample low and force-sample errors and slow requests.
- Profiling in production is the only place the answer is true, and it costs some CPU and carries a small operational risk. Sampling profilers are the compromise most services should take (Performance Testing a Backend).
- Chasing p99 for an internal admin tool is a real cost with little return. Decide which percentile has a user behind it.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe decomposition — queue, schedule, compute, wait, serialize — holds for any request-serving process.
- RUNTIME-SPECIFICThe "am I CPU-bound" signal differs sharply: Node needs event-loop lag because one thread runs all handlers, so 100% of one core can coexist with an idle-looking host; the JVM needs GC pause and thread-state sampling; CPython under a pre-fork server needs per-worker busy ratio, because a busy worker cannot accept anything else.
- DATABASE-SPECIFICWhat "the query got slow" means depends on the engine: Postgres plan changes follow autovacuum and statistics, MySQL/InnoDB has different index-selection behaviour, and a managed serverless database can add cold-start and connection-proxy latency neither of them has.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — System Design — deciding what latency the product actually needs before deciding which percentile to defend.