When the Trace Runs Out of Answers
The trace says pricing-service spent 240 ms and has no children. That is where tracing stops and profiling starts: one tells you which process is expensive, the other tells you which function inside it is.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Two questions, two tools, one handoff
Tracing and profiling are often presented as competing "pillars", which obscures the fact that they answer questions at different scales. A trace decomposes one request across processes: which service, which query, which external call. A profile decomposes one process across requests: which function, which call stack, which allocation site. Neither can do the other's job — a trace cannot see inside a function, and a profile has no idea which request it was serving.
The handoff between them is the most useful workflow in performance debugging, and it runs in one direction. Start with metrics (something is wrong), move to traces (it is wrong *here*), then move to profiles (it is wrong *because of this code*). Skipping to a profile first means profiling whichever service you suspected, which is how teams spend an afternoon optimizing a function on a branch with 80 ms of slack (The Critical Path Is the Only Path That Pays).
The handoff has a precondition worth stating: the fat leaf must be genuine local work. A wide childless span in a service that makes network calls is more likely a missing instrumentation than slow code (Carrying the Trace Across the Gap), and profiling it will show a stack sitting in epoll_wait — waiting, not working. Establishing CPU-bound versus I/O-bound before profiling saves the wasted session (Computing or Waiting?).
| Trace | Profile | |
|---|---|---|
| Unit of analysis | One request, across processes | One process, across requests |
| Answers | "Which hop consumed the budget?" | "Which function consumed the CPU?" |
| Blind to | Anything inside a leaf span | Which request or user it was for |
| Attribution | Exact, per request | Statistical, from stack samples |
| Typical overhead | Per-span cost, mitigated by sampling | Usually 1–3% at low sample rates |
| Fails when | Context propagation breaks | The process is waiting, not computing |
What a profiler actually records
Nearly all production profilers are sampling profilers: many times a second, the profiler interrupts and records the current call stack of each running thread. After ten seconds at 100 Hz you have roughly a thousand stack samples per thread; the fraction of samples containing a function is an estimate of the fraction of time spent in it. Nothing is counted exactly, which is the entire reason the overhead is affordable.
The statistical nature has two consequences people misread. First, small differences are noise: a function at 5.1% versus 4.4% in two profiles has told you nothing without far more samples. Second, a function that runs rarely but for a long time and a function that runs constantly but briefly can produce identical profiles — the profile measures where time went, not why, and the "why" often needs the trace you came from.
The alternative, instrumenting profilers, count every call exactly by injecting bookkeeping into each function entry and exit. They are precise, they are far too expensive for production, and their overhead distorts exactly the small hot functions you are trying to measure. They belong in local investigation, not in a running service.
| Profile | Records | Use when |
|---|---|---|
| CPU / wall | Stack samples on a timer | CPU is high, or a span is fat with no children |
| Allocation | Stacks at allocation sites, with bytes | GC pressure, allocation rate, memory churn (Allocation Rate Is a Cost Even Without a Leak) |
| Heap / retained | What is still reachable at snapshot time | Memory grows and does not come back (Memory Leaks: Growth That Does Not Come Back) |
| Lock / contention | Stacks blocked waiting on a lock | CPU idle, latency high, threads waiting on each other |
| Off-CPU / wall-clock | Where threads block, not just where they compute | I/O-bound work a CPU profile shows as nearly empty |
Profiling the right process at the right moment
Two operational details decide whether a profile is usable. Which instance: in a fleet of forty pods, profiling a random one during a tail-latency problem will usually profile a healthy pod, because the problem is by definition rare. Profile the instance the slow traces point at, or run continuous profiling so the window already exists (Always-On Profiling, and the Diff That Finds Regressions).
And which window: a profile averaged over five minutes that contains ten seconds of pathology shows the pathology at 3% and buries it under normal work. Narrow the window to the incident, or compare a profile from the bad window against one from a good window and read the difference rather than the absolute values.
For request-scoped questions, some runtimes support attaching profile data to a span or profiling only while a specific request is in flight. Where available this closes the loop completely — the trace points at the span, the span carries the stacks — and it is worth knowing whether your stack supports it before the next incident.
1# during a tail-latency incident2kubectl exec pricing-7f9c-xxxxx -- profile --seconds 3003 4# Problems:5# - random pod: p99 pathology may not be happening here at all6# - 5-minute window: 10 s of badness averaged into 290 s of normal7# - no baseline: 12% in serialize() -- is that high? Compared to what?1# 1. find the instance from the slow traces2# trace -> span.attributes["host.name"] = pricing-7f9c-q4m23 4# 2. narrow the window to the pathology5profile --target pricing-7f9c-q4m2 --seconds 30 --out bad.pb6 7# 3. baseline the SAME instance during normal traffic8profile --target pricing-7f9c-q4m2 --seconds 30 --out good.pb9 10# 4. read the DIFFERENCE, not the absolute percentages11profile diff good.pb bad.pb12# +41% calculateScore -> normalizeWeights -> sortByRelevanceThe second version answers "what is different when it is slow", which is the actual question. Absolute percentages in a single profile tell you what the service spends time on, which is a different and much less urgent question.
Key points
- Traces decompose one request across processes; profiles decompose one process across requests — the handoff runs metrics → traces → profiles, in that order.
- A fat leaf span is the signal to profile, but only after confirming the work is CPU-bound rather than waiting.
- Production profilers sample stacks on a timer, so attribution is statistical: small percentage differences are noise, not findings.
- Pick the profile type for the question — CPU, allocation, heap, lock contention and off-CPU answer different things and one of them is usually wrong for your symptom.
- Profile the instance the slow traces point at, over a window narrowed to the pathology, and read a diff against a healthy baseline rather than absolute percentages.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Metrics → engineer: endpoint p99 up, which says something is wrong but not where.
- 2Trace → engineer: one 240 ms
pricing-servicespan with no children, so the cost is inside that process. - 3CPU utilization → engineer: the process is at 94% of one core during the window, confirming compute rather than waiting.
- 4CPU profile diff → engineer:
calculateScoregrew from 9% to 55% of samples between the good and bad windows. - 5Code → engineer: a sort inside a per-item loop, quadratic in the result set that recently grew (Algorithmic Cost in a Request Handler).
- • "The profile shows 55% in
calculateScore, so that function is the bug." It is where time goes, not necessarily where the defect is — the caller invoking it 100× more often is a different bug with the same profile. - • "CPU profile is nearly empty, so the process is fine." An I/O-bound process has an empty CPU profile while being extremely slow. You need off-CPU or wall-clock profiling.
- • "This function is 5% in production and 4% in staging, so we regressed." Sampling noise. Compare distributions with enough samples before claiming a difference.
- • "We profiled the service and found the hot function." On a branch with slack, the hot function costs users nothing (The Critical Path Is the Only Path That Pays).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Take the fat span's host attribute from the trace and profile that instance specifically, not a random pod.
- • Capture two profiles of equal duration — one during the slow window, one during normal traffic on the same instance — and diff them.
- • Check CPU utilization of the process first: near-saturated means a CPU profile is the right tool; near-idle means you want off-CPU or lock profiling.
- • Record sample count alongside the profile; below a few hundred samples, percentage differences are not interpretable.
- • Confirm CPU-bound before profiling for CPU; otherwise switch to off-CPU or lock profiling and save the session.
- • Profile the right instance and window, and diff against a baseline — this is a process fix that improves every future investigation.
- • Once the hot stack is identified, check the call count as well as the cost per call: N × cheap and 1 × expensive have identical profiles and opposite fixes.
- • Adopt continuous profiling so the "capture a baseline" step is already done before the next incident ([[continuous-profiling]]).
- • After the code change, re-profile the same instance over a comparable window: the hot stack should fall to its baseline share.
- • The fat span in the trace should shrink correspondingly — a profile improvement with an unchanged span means you optimized something off the request path.
- • Endpoint p99 must move; the profile is evidence about the mechanism, not proof of user impact ([[measure-before-optimizing]]).
- • Confirm CPU utilization of the process fell, so the saving is real rather than moved to another thread.
- • Profiling adds overhead — small at production sample rates, but not zero, and it is highest on exactly the hot paths you care about.
- • Profiles can contain sensitive data in stack arguments or symbol names in some runtimes, so access needs the same care as logs.
- • Continuous profiling costs storage and a retention policy, and is one more pipeline to operate.
- • Narrowing a profile window sharpens the signal and increases the chance of missing an intermittent pathology entirely.
- • Keep continuous profiles retained long enough to diff across deploys, which turns "did this release regress CPU" into a query.
- • Add a CI benchmark for the specific hot path if it is algorithmically fragile (Regression or Tuesday? Telling a Real Change from Noise).
- • Alert on process CPU per request (CPU seconds ÷ requests) rather than raw CPU, so efficiency regressions surface even when traffic changes.
- • Record the profile and its diff in the incident review so the next responder starts from evidence (Debugging an Incident in Progress).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 240 ms span and 9% → 55% diff are constructed to show the workflow; real profiles are noisier and rarely this clean.
- RUNTIME-SPECIFICAvailable profile types, sampling mechanisms, symbolization quality and overhead differ substantially between runtimes and between JIT and AOT compilation.