Always-On Profiling, and the Diff That Finds Regressions
Profiling during an incident means capturing a baseline you do not have, on an instance that may be healthy, after the pathology has passed. Continuous profiling makes the baseline a query — and turns "did this release get slower" into a diff.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The baseline problem
Ad-hoc profiling has a structural flaw: you profile when something is wrong, which is exactly when you cannot get a clean comparison. The healthy profile you need for a diff is in the past, and nobody took it. So the investigation proceeds by reading a single profile and asking "does 15% in the serializer seem high?" — a question with no defensible answer.
Continuous profiling changes the shape of the problem by collecting low-rate profiles from every instance all the time, tagged with version, host, region and whatever else you can query by. The baseline is then whatever window you ask for: last Tuesday, the previous release, the pod that is not misbehaving. The investigation becomes a diff, which is a far easier question than an absolute judgement.
The second benefit is slower-burning and arguably larger: gradual regressions become visible. A 4% CPU increase per release is invisible in any single comparison and compounds to 27% over six releases. With a profile per version, that trend is a query, and the specific release and function are identifiable long after the fact.
What a diff makes obvious
The differential view answers the question people actually have. Not "what does this service spend CPU on" but "what is different". A table of frames sorted by change, or a flame graph coloured by delta, points at the regression directly — and it filters out everything that is expensive-but-normal, which is the majority of any profile and the main source of false leads.
It also catches the cases a single profile cannot. A function that stayed at 15% while total CPU per request rose 30% is not the problem, even though it is prominent. A function that went from 2% to 9% is the problem, even though it is small. Absolute prominence and change are different rankings, and only one of them is about the regression.
Version tagging is what makes this work across deploys, and it is worth being strict about: profiles tagged with the build identifier let you compare a canary against the stable fleet during a rollout, which turns a performance regression into something you catch before full deployment rather than after (Regression or Tuesday? Telling a Real Change from Noise).
| Function | v2.3 self | v2.4 self | Δ | Reading |
|---|---|---|---|---|
scoring.normalizeWeights | 2.1% | 9.4% | +7.3 | The regression. Small in absolute terms, large as a change. |
json.serialize | 15.2% | 15.0% | −0.2 | Prominent and unchanged — a false lead in a single profile. |
auth.verifyToken | 10.4% | 10.1% | −0.3 | Noise. |
log.write | 5.3% | 5.4% | +0.1 | Noise. |
(gc) | 4.0% | 7.8% | +3.8 | Secondary effect: the new code also allocates more. |
Cost, overhead and the things to watch
Overhead is the first question everyone asks and is usually the smallest problem: at typical production sampling rates (around 100 Hz per thread, or lower), CPU overhead is commonly in the low single digits, which is why running it everywhere is viable. It is not free, though, and it is highest on the hottest paths — the same paths you are measuring — so a very latency-sensitive service deserves a measured comparison rather than a shrug.
Storage and retention are the real operational costs, and they follow the same logic as Sampling Without Throwing Away the Evidence: keep recent profiles densely, older profiles sparsely, and per-release profiles for as long as you might need to bisect a gradual regression. A retention policy that keeps two weeks makes the six-release regression above undiagnosable.
Two things to watch. Symbolization must work for production builds, or the profiles you carefully retained are full of addresses when you need them. And profile data can be sensitive in runtimes where argument values or symbol names leak business logic — access controls comparable to logs are appropriate, and the same reasoning as What You Just Wrote Into a Log Half the Company Can Read applies.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| profiler CPU overhead | 1.4% | Within budget for always-on collection | normal |
| instances reporting | 38 / 40 | Two pods not reporting — blind spots during an incident | suspect |
| symbolized frame ratio | 61% | Nearly 40% of frames unreadable; profiles are half-useless | smoking gun |
| retention, per-release profiles | 14 days | Too short to bisect a gradual multi-release regression | suspect |
| profile ingest lag | 45 s | Acceptable — profiles are for analysis, not alerting | normal |
Key points
- Ad-hoc profiling cannot produce the baseline it needs, because the healthy window is in the past and nobody captured it.
- Continuous profiling makes the baseline a query: compare release to release, pod to pod, or window to window.
- A differential profile ranks by *change*, which separates the regression from the expensive-but-normal frames that dominate any single profile.
- Gradual regressions — a few percent per release — are invisible individually and only findable with retained per-version profiles.
- Overhead is usually low single digits; the real costs are storage, retention policy and keeping symbolization working for production builds.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Release → code: a scoring change adds per-item normalization work that is small relative to the whole request.
- 2Code → CPU: CPU seconds per request rises 4%, well inside normal variance for any single release.
- 3Six releases → fleet: the compounding increase reaches ~27%, and the instance count needed for the same traffic grows with it.
- 4Differential profile → engineer: comparing v2.3 with v2.4 shows
normalizeWeightsat +7.3 points, identifying both the release and the function.
- • "The profile shows serialize at 15%, that is our biggest problem." It was 15% before the regression too. Prominence is not change.
- • "Overhead makes continuous profiling unsafe for production." At typical sampling rates it is low single digits; measure it for your service rather than assuming either way.
- • "We have profiles, so we can bisect." Only within the retention window and only if they are symbolized and version-tagged.
- • "CPU per request is up but no single function grew much." Diffuse growth is a real pattern — often GC or a framework upgrade — and the GC frames in the diff usually say which.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Track CPU seconds per request per release as the headline efficiency number; the profile explains changes in it.
- • Diff profiles between the current and previous release rather than reading either one in isolation.
- • Monitor the fraction of instances reporting profiles — blind spots are silent until the incident where you need that pod.
- • Watch symbolized frame ratio; unsymbolized profiles are stored cost with no analytical value.
- • Deploy continuous profiling fleet-wide with version, host and region tags so every future comparison is a query.
- • Set retention by class: dense recent, sparse historical, and one retained profile per release for bisecting slow regressions.
- • Fix symbolization for release builds before you need it — an unreadable retained profile is worse than none, because it feels like preparation.
- • Wire a canary-versus-stable profile diff into the release process so regressions are caught during rollout ([[performance-regression-detection]]).
- • Confirm you can produce a differential profile between the last two releases in under a minute — that is the capability being bought.
- • Check instances-reporting coverage is effectively complete, since a missing pod is exactly the one that will misbehave.
- • Verify symbolized frame ratio is high on a production build, not just locally.
- • Re-derive the historical regression from retained data as a test of the retention policy.
- • Always-on collection costs a few percent of CPU on every instance, which is a real fleet-wide cost at scale.
- • Long retention costs storage and needs a policy; keeping everything forever is not a plan.
- • Profiles can expose internal structure and occasionally sensitive values, so they need access controls like logs.
- • A profiling agent is another component in the critical path of every process, with its own failure modes and upgrade cycle.
- • Alert on CPU seconds per request per release, which is the metric the profiles exist to explain.
- • Alert on profiler coverage dropping below a threshold, so blind spots surface before an incident.
- • Make a canary profile diff part of the deploy checklist rather than something people remember to do.
- • Review retention against the longest regression you have ever had to bisect, and set the policy from that.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe differential table and the overhead and coverage figures are constructed to show the workflow and the readings.
- RUNTIME-SPECIFICAchievable overhead, symbolization quality and available profile types vary widely by runtime and profiler implementation.