Debug Infotarget

Symbolication

Turning `0x00007f8a3c0012ef` back into `parse_header at http.c:184`, using symbols and debug information you deliberately stripped out of the shipped binary. The whole thing works or fails on one detail: whether the build ID ties the two artifacts together.

The question

My crash report is a list of hex addresses. How do I get function names and line numbers back?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute

What the program is here

Three artifacts that must be reunited. The stack trace: a list of runtime addresses, captured on a machine that may have had address-space layout randomization and shared libraries mapped anywhere. The shipped binary: stripped, containing the instructions and a build identifier. The symbol file: the debug information that was split out at build time. Symbolication is the process of computing file offsets from runtime addresses and looking them up — and the representation that makes it possible is the build identifier, which is the only thing linking a crash to the exact bytes that produced it.

What this phase may assume or do

Symbolication is a read-only lookup, so its precondition is entirely about correspondence rather than behavior: the symbol file must have been produced from the identical compilation of the identical source, matched by a cryptographic or unique build identifier rather than by version number, path or timestamp. Symbolicating against a rebuild of the same source is not valid — a different inlining decision or a different section layout shifts addresses, and the result is a stack trace that is plausible and wrong. That is worse than no symbolication at all, and it is why matching by identity rather than by name is the load-bearing rule.

Key points

Symbolication converts runtime addresses into function names, files and line numbers using symbols kept outside the shipped binary.
It is three lookups: address to module and offset, offset to function, offset to source line plus inlined frames.
The load base must be subtracted first; ASLR means a runtime address means nothing without knowing where the module was mapped.
Build IDs — GNU build ID, Mach-O LC_UUID, PDB GUID and age — are what tie a crash to the exact bytes that produced it.
Never match symbols by version, path or filename: a rebuild can shift addresses and produce a plausible, wrong trace.
Inlined frames are recovered only if the tool is asked (addr2line -i); omitting it silently drops frames and misattributes the fault.
The pipeline is build with -g, split, strip, archive by build ID, capture module bases at crash time, symbolicate server-side.
The most common failure is symbols that were never archived, discovered on the day they are needed.
Source maps, ProGuard mapping files and .dSYM bundles are the same pattern in different containers.

What has to be recovered, and from where

A crash in production yields addresses. Turning them into something actionable is three lookups, and each has its own failure mode.

Address to module. The process had several mapped regions — the executable and every shared library — each loaded at an address chosen at runtime. Subtracting the module's load base from the runtime address gives an offset that means something in the file. Getting this wrong is the most common cause of a symbolicated trace that is confidently, uniformly wrong: every frame resolves to a plausible function that is not the right one.

Offset to function. The symbol table maps address ranges to symbol names. This alone gets you a readable trace, and it is what a stripped-but-not-fully-stripped binary provides. In C++ and Rust the name is mangled and has to be demangled to be legible — see [[name-mangling]].

Offset to source line, plus inlined frames. The line table gives file and line; the inline records reconstruct the frames that inlining erased. Skipping the inline step silently drops frames, which is the second most common defect in a symbolication pipeline and produces traces that blame the wrong function entirely.

The tools that do this are small and specific: addr2line -f -C -i, llvm-symbolizer, atos on macOS, dia2dump or the DIA SDK on Windows. All of them need the debug information, and none of them can do anything with a stripped binary and no symbol file.

From a raw address to a frame, one step at a time

1# 1. What was in the process, and where was it mapped?
2#    (from /proc/PID/maps at crash time, or the core dump's notes)
37f8a3c000000-7f8a3c04a000 r-xp  /usr/lib/libhttp.so.1
4 
5# 2. Runtime address 0x7f8a3c0012ef, load base 0x7f8a3c000000
6#    -> file offset 0x12ef
7 
8# 3. Which build is this? The build ID, from the module in the dump:
9readelf -n /usr/lib/libhttp.so.1
10  Build ID: 3f5a9c1e0b7d4a2f8e6c1d9b0a4f7e2c5d8b3a61
11 
12# 4. Fetch the matching symbols from the symbol store, then:
13addr2line -e libhttp.so.1.debug -f -C -i 0x12ef
14  read_field                         <- inlined frame, recovered by -i
15  /src/http/parse.c:57
16  parse_header
17  /src/http/http.c:184
18 
19# Without -i you get only `parse_header` and never learn that the
20# fault was in `read_field`, which is where the bug is.

Every step can fail silently and produce output that looks fine. A wrong load base gives plausible names from the wrong addresses. A mismatched build ID gives plausible names from the right file compiled differently. A missing -i gives a shorter trace with no indication anything was dropped. None of these produce an error message, which is why a symbolication pipeline needs assertions rather than eyeballs.

Build IDs are the whole mechanism

The thing that makes any of this reliable is a unique identifier stamped into the binary at link time and recorded in the symbol file, so that "the symbols for this exact binary" is a lookup rather than a guess.

On ELF this is the GNU build ID: a hash written into a .note.gnu.build-id section by ld --build-id, readable with readelf -n. The convention is to store symbol files at /usr/lib/debug/.build-id/3f/5a9c1e....debug, which is what gdb searches automatically, and what debuginfod serves over HTTP so a debugger can fetch symbols for a distribution binary on demand. On Mach-O it is the LC_UUID load command, matched against the UUID recorded in the .dSYM bundle — dwarfdump --uuid prints both. On Windows the PDB is identified by a GUID plus an age counter, both stamped into the executable's debug directory, which is exactly what a Microsoft-style symbol server indexes.

The rule that follows is worth stating as a rule: never match symbols by version number, path or filename. A rebuild of the same source with the same compiler and the same flags can differ — different absolute paths, a different build timestamp, a different link order, a nondeterministic hash seed — and the resulting addresses shift. Symbolicating against a near-miss produces a trace that reads correctly and points at the wrong lines, and there is no signal to tell you. This is also the practical reason [[reproducible-compilation]] matters beyond supply-chain integrity: a reproducible build lets you regenerate symbols you failed to archive and prove they correspond.

Identity and symbol storage, per platformtarget

Platform	Identifier	Symbols live in	Fetched by
Linux / ELF	GNU build ID in `.note.gnu.build-id`	A separate `.debug` file, or `.dwo` with split DWARF	`/usr/lib/debug/.build-id/`, or debuginfod over HTTP
macOS / Mach-O	`LC_UUID`	A `.dSYM` bundle built by `dsymutil`	Spotlight index, or an explicit path passed to `atos`
Windows / PE	PDB GUID plus age, in the debug directory	A separate `.pdb` file, always	A symbol server keyed by GUID and age
JavaScript	A release identifier you choose	A `.map` file — see `[[source-maps]]`	Uploaded to the error reporter per release
JVM / .NET	Class file or assembly metadata	Line tables inside the artifact itself	Nothing to fetch; it is already there

The pipeline, and where it breaks

A working production symbolication setup is five steps, and every one of them is a place teams lose the ability to read their own crashes.

Build with debug information, at production optimization. -O2 -g — see [[debug-vs-release]]. If this step is skipped, nothing downstream can help.

Split and strip. Extract the debug information to a companion file, strip the binary, ship the stripped one. The shipped artifact is the same size as a no-debug build.

Archive the symbols, keyed by build ID, in a symbol server or an artifact store, with a retention policy at least as long as the binary can plausibly still be running somewhere. This is the step that is skipped most often, and its absence is discovered only when a crash arrives.

Capture enough at crash time. The addresses alone are not enough: you need the module list with load addresses and build IDs. A minidump or core dump contains this; a naive backtrace() call does not. Crash handlers such as Breakpad and Crashpad exist largely to capture this correctly from inside a process that is already broken.

Symbolicate server-side. Do it in the reporting service, not on the user's machine — which requires no symbols on the device, keeps them private, and lets you re-symbolicate old crashes when you improve the pipeline.

The two failure modes worth naming explicitly, because they are silent: symbols not archived, discovered on the day of the incident; and symbols archived but not matched by build ID, producing traces that are subtly wrong in a way nobody detects until a fix lands in the wrong function.

Assert in CI that the shipped binary has a build ID and that a symbol file with that ID reached the store.
Symbolicate a known crash as part of the release pipeline; a smoke test for symbolication catches the whole failure class.
Always pass the inline flag (addr2line -i, llvm-symbolizer --inlines), or accept silently missing frames.
Retain symbols for at least as long as any build can still be deployed, which for mobile means years.
Keep frame pointers (-fno-omit-frame-pointer) if your crash handler or profiler walks stacks naively.
Record the load base and build ID of every module at crash time, not just the addresses.

The same problem everywhere else

Once the shape is clear — strip the human-readable information from the shipped artifact, keep it somewhere indexed by an immutable identifier, rejoin them at diagnosis time — you see it in several places that do not look related.

Minified JavaScript and [[source-maps]] are the identical pattern with a JSON container and a release identifier instead of a build ID. Android's ProGuard and R8 produce a mapping file that retrace applies to obfuscated stack traces, for the same reason and with the same "did you keep the mapping file" failure. iOS crash reports need the .dSYM matched by UUID. Go binaries carry a symbol table by default and are readable without any of this, which is a deliberate design choice with an artifact-size cost. WebAssembly has its own DWARF-in-a-custom-section arrangement.

The generalisation worth carrying: shipping a small artifact and shipping an unreadable one are different decisions that most default build configurations conflate. Splitting is what separates them, and the identifier is what makes splitting safe.

How it works

The steps, in the order the compiler takes them.

The linker stamps a unique build identifier into the binary and the same identifier into the debug information extracted from it.
The shipped binary is stripped of symbols and debug sections; the symbol file is uploaded to a store indexed by that identifier.
At crash time a handler captures the faulting thread's stack — by walking frame pointers or by interpreting unwind tables — plus the list of loaded modules with their load addresses and build identifiers.
The report is transmitted with addresses, module bases and identifiers, but no symbols.
The reporting service subtracts each module's load base from each address to obtain a file offset.
It fetches the symbol file matching that module's build identifier from the store, refusing to proceed if there is no exact match.
It looks the offset up in the symbol table for a function name, and in the line table for a file and line.
It expands inline records so that each address yields the full chain of inlined functions rather than only the outermost one, and demangles the resulting names.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

A crash dashboard shows thousands of reports grouped by hex address, because symbols were never archived for the release that is failing.
A symbolicated trace names functions that are not on the call path, because the load base was wrong or the module list was not captured.
A trace resolves against a rebuild of the same source and points at lines a few off, sending the investigation to the wrong branch of the wrong function.
The reported crashing function is a large caller and the actual fault is in a small inlined helper, because the symbolizer was not asked for inline frames.
Symbol upload silently fails in CI for two weeks and nobody notices until an incident, because nothing asserted that the upload succeeded.
Symbols were retained for ninety days and a crash arrives from a client on a year-old version, which is now permanently unreadable.
A stack trace has two frames because the crash handler walked frame pointers in a binary built with -fomit-frame-pointer and gave up immediately.

When it helps

Any software deployed to machines you do not control: desktop applications, mobile apps, agents, embedded devices, and anything a customer runs.
Server-side crash and panic reporting, where the alternative is a log line containing addresses and no way to act on it.
Continuous profiling in production, which uses the same symbol lookup to attribute samples to source lines rather than to addresses.
Post-mortem analysis of a core dump, which is only interpretable with symbols matched to the exact binary that produced it.

When it hurts

When symbol storage is treated as free: full debug information for every build of a large application, retained for years, is a real and growing storage bill that needs a policy.
Where the symbols themselves are sensitive — function names and file paths reveal internal structure — and the store is not access-controlled.
In environments where crashes cannot be transmitted at all, in which case the effort belongs in on-device logging instead.
When it creates false confidence: a symbolicated trace is only as trustworthy as the build-ID match and the inline expansion behind it.

What it costs

Every one of these is paid by something.

Archiving full debug information buys readable crashes for every deployed build and pays storage that grows with release frequency times artifact size, plus a retention policy someone must own.
Server-side symbolication buys private symbols and re-symbolicatable history and pays an infrastructure dependency in the crash path.
Client-side symbolication buys a working trace with no server round trip and pays by shipping symbols to the device, which is both size and disclosure.
Keeping frame pointers buys reliable stack capture in a crash handler and profiler and pays a register and a small performance cost.
Line-table-only debug information (-g1) buys most of symbolication's value at a fraction of the storage, and pays the ability to inspect variables in a core dump.
Strict build-ID matching buys correctness and pays unreadable crashes whenever the archival step failed — which is the correct failure direction, since a plausible wrong trace costs more than an unreadable one.

What else you could do

What a different compiler or language does instead, and when that is better.

Ship symbols with the binary. Simplest possible arrangement, no store to operate, and it costs artifact size and publishes your internal structure.
Ship a symbol table without debug information: function names but no line numbers, at a small fraction of the size. Often the right point for constrained targets.
Log structured context instead of relying on stack traces — a language-level error with a message and fields is readable without any symbol infrastructure, which is why managed and Go services frequently need less of this.
Use a runtime that carries symbols intrinsically: JVM and .NET keep line tables in the artifact, Go embeds a symbol table by default, and Python tracebacks are source-level by construction. The cost is artifact size and the inability to strip.
Reproducible builds as a fallback: if the build is bit-for-bit reproducible, symbols can be regenerated from source for a build whose symbols were lost, and proved to correspond — see [[reproducible-compilation]].

See it for yourself

The flag, dump or tool that shows you this directly.

Linux: addr2line -e ./bin -f -C -i 0xADDR (function, demangled, with inlines), llvm-symbolizer --inlines --demangle < addresses.txt, eu-unstrip to recombine a stripped binary with its symbols.
Find and check identity: readelf -n ./bin prints the build ID; file ./bin reports whether it is stripped and shows the build ID; objcopy --add-gnu-debuglink records the association.
Fetch symbols on demand: debuginfod-find debuginfo <build-id>, and set DEBUGINFOD_URLS so gdb resolves distribution binaries automatically.
macOS: atos -o App.app.dSYM/Contents/Resources/DWARF/App -arch arm64 -l 0x100000000 0x100001234; dwarfdump --uuid on both the binary and the .dSYM to confirm they match before trusting anything.
Windows: llvm-pdbutil dump --symbols app.pdb, or !analyze -v in WinDbg with the symbol server configured via _NT_SYMBOL_PATH.
Android and JavaScript: retrace mapping.txt trace.txt for R8/ProGuard, ndk-stack -sym obj/local/arm64-v8a for native crashes, and sentry-cli sourcemaps upload --release <id> for the JavaScript equivalent.
Prove your own pipeline works: deliberately crash a release build, feed the report through symbolication, and assert the result names the right function. Do it in CI, not by hand once.

Plausible wrong readings

Stated the way a confident engineer states them.

"We ship stripped binaries, so we cannot symbolicate." Stripping is exactly the intended arrangement — provided the symbols were archived first and can be found by build ID.
"I can rebuild from the same tag to get symbols." Only if the build is reproducible. Otherwise the addresses shift and you get a trace that reads correctly and points at the wrong lines.
"The stack trace is complete." Not unless inline frames were expanded. A missing -i drops exactly the small hot functions where faults tend to be.
"Symbolication is a debugger feature." It is a build-and-release pipeline. The tool that does the lookup is trivial; keeping the symbols and matching them is the entire problem.
"Addresses in a crash report are meaningful." They are meaningful only relative to a module load base that ASLR chose at runtime. Without the module map they resolve to nothing.

Misconceptions

The claim, and what is actually true.

Symbolication requires an unstripped binary.

It requires the *symbols*, which can and should live in a separate file. That is exactly why stripping and symbolication are compatible.

A symbolicated trace is trustworthy by virtue of being symbolicated.

Its accuracy depends on the build-ID match, the load-base arithmetic and inline expansion. Any of the three being wrong produces a readable trace pointing at the wrong code.

Keeping symbols is a debugging nicety.

For software on machines you do not control it is the primary diagnostic channel. Discarding it converts every field crash into an unactionable report.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

The binary you ship has had the names and line numbers stripped out to keep it small, so when it crashes on a user's machine the report is a list of hex addresses. Symbolication is putting them back: subtract where each library happened to be loaded, look the resulting offset up in the debug information you saved at build time, and get a function name and line. The one thing that makes it reliable is a unique identifier stamped into the binary at link time — the build ID — which lets the tooling fetch the symbols for exactly this build rather than approximately this build.

practical

Set up five things and check them in CI. Build with -g at production optimization. Split the symbols out and strip the binary. Upload the symbol file keyed by build ID, and assert that the upload happened. Make the crash handler capture module load addresses and build IDs, not just stack addresses. And always pass the inline flag when symbolicating, or you will silently lose the frames where the bug usually is. Then deliberately crash a release build and symbolicate it as a smoke test — that one test catches every way this pipeline breaks, and it breaks quietly.

advanced

The general pattern is worth naming because it recurs: ship the minimum, retain the human-readable half, and bind the two with an immutable identifier. Source maps do it for JavaScript, ProGuard mapping files for Android, .dSYM bundles for Apple platforms, and DWARF plus a build ID for native Linux. The identifier is what makes the arrangement safe — matching by version or path admits a near-miss, and a near-miss produces a trace that is confidently wrong, which costs more than an unreadable one. That is also why [[reproducible-compilation]] is more than a supply-chain concern: a bit-reproducible build lets you regenerate symbols you failed to archive and *prove* they correspond, converting the worst failure mode in this pipeline from permanent to recoverable. The connection is not obvious from either side, and it is the strongest practical argument for reproducibility in an organisation that does not otherwise care about it.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

targetIdentifier and container are platform-specific in every particular. ELF uses a GNU build ID in a note section with symbols in a separate .debug file; Mach-O uses LC_UUID with a .dSYM bundle; PE/COFF uses a PDB GUID plus an age counter with an always-separate .pdb. The conventional search paths differ too — /usr/lib/debug/.build-id/ on Linux, Spotlight on macOS, _NT_SYMBOL_PATH on Windows. The pattern transfers everywhere; none of the mechanics do.

implementationWhether a rebuild of identical source yields identical addresses depends on the toolchain and the build environment: absolute paths, timestamps, link order, parallel codegen unit assignment and hash seeds can all vary. -ffile-prefix-map, SOURCE_DATE_EPOCH and deterministic archive flags exist to remove these sources of variation, and a build that has not been checked for reproducibility should be assumed not to be. Matching by build ID rather than by source revision is the defence that does not depend on any of it.

typicalThe pipeline described — build with symbols, strip, archive by build ID, symbolicate server-side — is standard practice for crash reporting across desktop, mobile and server software, and it is what Breakpad, Crashpad, Sentry and platform crash reporters all assume. It is not required: Go ships symbol tables in the binary by default and needs none of it, and managed runtimes carry line information in the artifact. The pattern applies wherever the shipped artifact is stripped, which is most native software.

If you were asked this in an interview

Walk me through turning a hex address in a production crash report into a file and line.
Why must symbols be matched by build ID rather than by version number?
A symbolicated trace names a large function, but the bug is in a small helper. What went wrong?

Connections

Performanceflame-graphs cpu-profiling incident-debugging continuous-profiling

Securityartifact-integrity supply-chain-security

Cloudartifact-promotion

Backendproduction-debugging

OS & Networkingdebugging-tools process-memory-layout memory-mapping virtual-address-space

Domains that do not exist yet

DevOps / Production Engineering — Operating a symbol store: upload, indexing, retention and access control
Every common failure in this lesson is operational rather than technical — symbols never uploaded, uploaded but unindexed, or retained for less time than the software is deployed. Running that store and its retention policy is a delivery-engineering responsibility owned there; why the identifier must be a build ID and what a mismatch costs is ours.
Testing & Reliability Engineering — Crash reporting and grouping as a reliability signal
Symbolication feeds a triage process: grouping crashes by normalised stack, prioritising by affected users, tracking regressions across releases. That process and its metrics are owned there; the correctness of the trace it groups on, and the ways it can be silently wrong, are ours.

Runtime gain1

Compile cost2

Debuggability5

Implementation2

Context dependence5

Symbolication has no runtime cost — it happens after the fact, off the machine that crashed. The compile-cost rating covers generating and splitting the debug information that feeds it. Debuggability is maximal because for software running on machines you do not control, this is frequently the only diagnostic signal there is. Implementation complexity is low: the lookup tools already exist and are small. Context dependence is maximal, since the identifier, the symbol container, the store and the fetch mechanism are entirely platform-specific.

Try it

linker loader →assembly explorer →