Source Locations

A diagnostic points at something: main.rs, line 12, column 9. That looks like three or four independent facts, and storing them per node would be the obvious design. It is not the design anyone uses.

The reason is arithmetic. Compilers compare, merge, sort and nest positions constantly — is this token inside that macro expansion, does this span contain the cursor, what is the union of these two ranges — and every one of those operations is a single integer comparison on offsets and a mess on (line, column) pairs. A position is therefore one number, and rustc goes further: it uses one global offset space across every file in the crate, so a single u32 identifies the file *and* the position within it, by binary searching a table of per-file start offsets.

Line and column are computed only when a diagnostic is being rendered. Each source file carries a sorted array of the byte offsets at which lines start; a binary search over it converts an offset to a line number, and the column is the difference from that line's start — counted in whichever unit you have decided to count in, which is the next section and the one that causes the trouble. See [[binary-search]] in DSA for the lookup, and note that this is one of the few places in a compiler where a plain sorted array beats every fancier structure, because it is built once and read many times.

Column 9 of what? The answer is not obvious and there are four defensible ones, and the tools you use every day do not agree.

Take the source let x = "café";. It is sixteen bytes, fifteen characters and fifteen UTF-16 code units, because é is two bytes in UTF-8 but one of everything else. A compiler storing byte offsets records the string literal as [8,15) — seven bytes. An editor counting UTF-16 units, which is what LSP specifies by default, records it as [8,14). Compute an underline length from the byte span and you draw seven carets under a six-column token. It is off by one, only on lines with non-ASCII content, and only in the part of your product that renders errors.

The spans in the device below are UTF-16 units, because that is what this page slices with — the same choice, and the same trap, one layer down. The three columns disagree exactly where you would expect, and the notes name the byte answer alongside.

The specific, real gotcha is the Language Server Protocol. LSP positions are (line, character) pairs where `character` is an offset in UTF-16 code units by default. Not bytes. Not Unicode scalar values. UTF-16 code units — the thing JavaScript's String.length counts — because the protocol was designed around VS Code, whose editor buffer is UTF-16.

Almost no compiler stores positions that way. rustc, Go, Clang and tree-sitter all work in UTF-8 bytes. So every language server contains a conversion layer, and every bug in that layer looks the same: everything works perfectly until a file contains an emoji or an accented character, after which every diagnostic on that line is offset, every rename edits the wrong range, and every hover is one character off. Astral-plane characters make it worse — an emoji is one scalar value, four UTF-8 bytes and two UTF-16 code units — so a line with one emoji shifts positions by different amounts under each convention.

LSP 3.17 added negotiation: a client advertises general.positionEncodings and a server may pick utf-8 or utf-32 instead. It is opt-in on both sides, so the UTF-16 default is still what you get unless both ends implement the negotiation.

Two more, both smaller and both real. Base: rustc, GCC and Clang display 1-based lines and 1-based columns; LSP uses 0-based lines and 0-based characters; some tools mix them, displaying 1-based lines with 0-based columns. There is no principle here, only convention, and the conversion is one addition in the place where the two meet — which is exactly the kind of code that gets it right for lines and wrong for columns.

Tabs: is a tab one column or eight? For an underline to line up under a line indented with tabs, the renderer must expand them the way the reader's terminal will, which it cannot know. rustc renders tabs as four spaces in its own output and adjusts the underline accordingly; GCC has -ftabstop=N for exactly this. Whatever the choice, the *stored* position must remain a byte offset — expanding tabs before storing turns the position into something that cannot be mapped back to the file.

And one thing that is not a convention but a rule: positions index the source buffer *as read from disk*, before any normalisation. Line-ending translation, BOM stripping, tab expansion, escape decoding and macro expansion all change the text; every one of them must either happen before offsets are assigned or maintain an explicit mapping. This is why rustc's SourceMap records the byte positions of any normalisation it performed, and why source maps exist at all for languages that are compiled from other languages — see [[source-maps]].

The cost of doing it properly is small and structural: one integer per position rather than three, a line-start table per file built once during lexing, a conversion at the rendering boundary, and a decision about units written down somewhere. Under a hundred lines of code, and it must exist before the first token is produced.

The cost of getting it wrong is that it cannot be fixed later. A frontend that stored line and column instead of offsets cannot cheaply add span merging. One that lost the original buffer cannot render an underline. One that normalised text before recording offsets has positions that are wrong by a per-file amount nothing can recover. Every diagnostic, every jump-to-definition, every rename and every source map in the product's future depends on a decision made in the lexer on day one — which is why this is a lesson and not a footnote, and why the pipeline explorer's ability to link a click in the assembly panel back to a character in the source panel is the whole argument made visible.