What a Profile Costs You

Without a profile the optimizer uses heuristics: loops are probably hot, error paths probably are not, small functions are probably worth inlining. These are weak and broadly unbiased — wrong in a diffuse, low-stakes way.

With a profile the optimizer stops hedging. It commits: this function gets a large inlining budget, this block goes in the hot section, this value wins the register, this other code goes to .text.unlikely where fetching it costs a miss. If the profile described the wrong workload, every one of those commitments is confidently wrong, and the cold section now holds code your users execute constantly. The failure is not a missing gain; it is a real regression relative to the heuristic build.

The realistic version of this is rarely a comically wrong workload. It is a training run that omits the paths that matter: profiling only warm-cache requests when production is dominated by cold-start work; profiling with a small dataset so a size-dependent branch never flips; profiling the happy path when a tenth of production traffic is retries and error handling. Each produces a binary optimized for a program adjacent to the one you ship.

The multi-modal case deserves its own mention because merging hides it. A service that spends mornings serving reads and nights running batch jobs has two hot paths; summing their counters produces a profile that describes neither, and the optimizer will lay out an average that is worse for both than optimizing for one would have been.

A profile is collected against one revision of the source. Every edit afterwards moves it further out of correspondence, and the decay is not uniform: it hits precisely the functions people are working on, which are disproportionately the ones being optimized for.

The mechanism that keeps this sound also makes it invisible. Counts are attributed per function via a hash of that function's control-flow structure. Edit the function so its structure changes, and the hash no longer matches, and the compiler correctly declines to apply the counts — because attributing them to shifted blocks would be worse than having none. What it does not do is fail the build. The function is simply compiled with heuristics, and nothing in the output says so unless you ask.

So the operational picture is a slowly widening gap between what the team believes is optimized and what is: the profile is still there, the flags are still on, the build is green, and an increasing fraction of the hot code is being compiled as if PGO were off. Teams that run PGO seriously monitor profile coverage — what fraction of executed functions matched — as a metric, not as a build check.

The pipeline is no longer source in, binary out. It is: build instrumented, run a workload, merge, build again. That middle step is a *process*, with a runtime, a dataset, an environment and a failure mode, sitting inside what used to be a pure function of the source tree.

The immediate consequence is for [[hermetic-compilation]]: the profile is an input that no source-control system holds and that nothing in the build declares. Two engineers on the same commit get different binaries if they trained differently. A cache key computed over the sources is now incomplete unless the profile is hashed into it — and once it is, regenerating the profile invalidates every artifact at once.

The disciplined answer is to make the profile an explicit, versioned artifact: check it in or store it in an artifact registry, address it by content hash, include that hash in the cache key, and regenerate it on a schedule with a recorded workload description. That restores reproducibility — the same sources plus the same profile give the same binary — without pretending the profile came from nowhere. What it does not restore is [[reproducible-compilation]] in the "anyone can rebuild this from the source tree" sense, because now they also need the profile artifact.

The second consequence is a new class of pipeline failure. A CI job that skips the training step, a workload that crashes halfway, a merge that produces an empty profile — all of these can produce a release build that silently applied no profile. This is worth a hard failure rather than a warning, because it is invisible in every other way.

AutoFDO and its relatives take a different route. Instead of building an instrumented binary, they sample an ordinary optimized binary already running in production — hardware performance counters, typically taken with last-branch-record support so that both instruction addresses and taken branches are captured. The samples are converted back into a source-level profile using the binary's debug information, and fed into the next build.

What this buys is the removal of every friction point above. There is no instrumented binary, so no unshippable slow build and no training run to operate. The overhead is small enough — on the order of a percent — to run continuously on real traffic, so the profile is representative by construction and never more than a day or so stale. For a service with continuous deployment, that combination is decisive.

What it costs is resolution and attribution. Samples are statistical, so rarely executed but important paths may be represented by very few samples or none. Attribution runs backwards through debug information, so it is only as good as the line tables — and after inlining, attributing a sample to the right *inlined instance* of a function requires the compiler to have emitted enough inline-frame information, which is exactly the information optimized builds are stingiest with. Getting a sample count for a specific control-flow edge, which instrumentation gives exactly, is inference rather than measurement here.

The honest summary is that instrumentation gives precise counts of a possibly unrepresentative run, and sampling gives approximate counts of a definitely representative one. In production systems the second is usually the better trade, and the reason is the recency point from the previous lesson: staleness costs more than resolution.