One Tree, Six Consumers
The compiler is no longer the only thing that parses your code. The formatter, the linter, the language server, the refactoring engine and the documentation generator all need the same tree — which is why a modern frontend is built as a library and why "just parse it again" is the wrong answer six times over.
Why do modern compilers ship their frontend as a library instead of just a binary?
The same AST, now with more than one reader. What changes is not the tree but its contract: it must be complete enough for a compiler, faithful enough for a formatter, resilient enough for an editor that sees broken code most of the time, and stable enough that external tools can be written against it. Those are four different requirements on one data structure, and reconciling them is what "frontend as a library" actually means.
A shared frontend may assume only what all its consumers can guarantee, which is much less than a batch compiler assumes. It may not assume the input parses — an editor sends incomplete code on every keystroke, so the tree must represent errors as nodes rather than as an exception. It may not assume a single-threaded reader, a single version of the file, or that analysis will run to completion. And it may not assume its own output is private: once tools depend on the node shapes, changing them is a compatibility break, which is a constraint the compiler-only version never had.
Key points
- Six different tools need the same answers about a program; independent parsers guarantee they disagree.
- A modern frontend is a library, and the compiler binary is one of its clients.
- Editors send code that does not parse, on every keystroke — so error recovery and error nodes are structural requirements, not politeness.
- Latency becomes a correctness property: an editor cannot wait for a whole-file reparse, which is what forces incrementality.
- The node shapes become a public API the moment external tools depend on them, and then they cannot change.
- The shared tree is bigger and more constrained than any single consumer needs, and everyone pays a little for someone else's requirement.
Six tools, one parse
For most of the history of compilers there was one consumer of the parse: the compiler. Editors did syntax highlighting with regular expressions, linters had their own parsers, formatters had a third, and each was subtly wrong about the language in a different way. Anyone who used a C IDE in 2005 remembers what that felt like.
That arrangement collapsed for a simple reason: the number of tools that need a correct answer about the program grew, and every independent parser is a place where the tool and the compiler disagree about what the code means. A linter with its own parser flags valid code. A formatter with its own parser mangles a construct it does not know. The only durable fix is that everyone reads the same tree, produced by the same code the compiler uses.
So a modern frontend is a library first and a command-line compiler second. The compiler binary is one client of it. That inversion is the single biggest structural change in compiler engineering in the last twenty years, and it explains a long list of otherwise puzzling design decisions in [[ast-node-design]] and [[ast-transformations]] — immutable trees, error nodes, spans on everything, no parent pointers, arena indices for cheap side tables.
| Consumer | Needs from the tree | Fails if |
|---|---|---|
| Compiler | Complete, checked, semantically annotated | Anything is missing — it must reject the program rather than guess |
| Formatter | Every token, every comment, every blank line | Trivia was discarded — it deletes the user's comments |
| Linter | Stable node kinds and resolved symbols | Node shapes change between versions — every rule breaks at once |
| Language server | A tree for code that does not parse, updated per keystroke | Parsing throws on error, or reparse is O(file) — the editor stutters |
| Refactoring engine | Symbols, references, and exact spans to edit | Resolution is approximate — the rename misses a use or renames the wrong one |
| Doc generator | Declarations, signatures and attached comments | Comments are not associated with the declarations they precede |
The requirement the compiler never had: broken code
A batch compiler sees a finished file. A language server sees const x = and is expected to offer completions for what comes next. That is not an edge case — it is the *normal* state of a file being edited, and it is the requirement that most changes a frontend's design.
It means parse errors cannot be exceptions. The parser must recover, insert error nodes where it could not build a real one, and hand back a tree that covers the whole file anyway. Every later phase must tolerate those nodes: name resolution must not crash on an error expression, the type checker must produce an error type that does not cascade into fifty spurious messages, and completion must work in the subtree that *did* parse. [[error-recovery]] and [[parser-synchronization]] stop being politeness features and become load-bearing.
It also means latency is a correctness property in a way it never was for a compiler. A batch compiler that takes 400ms per file is fine. A language server that takes 400ms per keystroke is unusable, which is why incremental reparse and query-level caching are not optimizations here — they are the reason the architecture exists.
What it costs to have six consumers
The benefits are obvious and the costs are the part worth learning, because they are what teams underestimate.
The node shapes become a public API. Once ESLint rules, Babel plugins and a thousand codemods are written against ESTree, the node shapes cannot change; a new language feature must be expressed in a way that does not break existing consumers, which sometimes means an uglier tree than the compiler alone would have chosen. Every mature ecosystem has at least one node kind that is shaped the way it is purely for backwards compatibility.
The tree grows to satisfy the union of requirements. The compiler does not need comments; the formatter does. The compiler does not need error nodes; the editor does. The compiler is happy with mutation; the language server is not. The result is a tree that is larger and more constrained than any single consumer would build, and every consumer pays a little for someone else's requirement.
And the release cadence couples. A frontend library that six tools depend on cannot make a breaking change casually, which slows the compiler team down in exchange for the ecosystem not fragmenting. That is a genuine trade, and it is the reason some projects deliberately expose a *stable* tree separate from their internal one.
- Node shapes become a compatibility surface: changing them breaks tools you do not own.
- The tree carries fields most consumers never read, and every consumer pays the memory.
- The frontend must be usable as a library — no global state, no
exit()on error, no writing diagnostics to stderr from inside a parse. - Diagnostics become structured data with spans and severities rather than formatted strings, because an editor needs to place them, not print them.
- Versioning and API stability become a compiler-team responsibility, which it never was when the only consumer was the code generator.
Where this goes next
Once the frontend is a library, the natural next step is to expose it over a protocol so that every editor gets every language without N times M integrations. That is [[lsp]], and the server on the other end of it is [[language-server]] — a compiler frontend that never finishes, never exits, and answers questions instead of producing an artifact.
The analyses that server runs are the same ones the compiler runs, asked differently: "what is the type here" is [[type-checking]] scoped to one node, "where is this defined" is [[name-resolution]] read backwards, "is this reachable" is [[control-flow-analysis]], and "is this dangerous" is [[static-analysis]]. The analysis module is where those get treated as first-class tools rather than as compiler internals — and the reason they can be is everything this module has been building toward.
How it works
The steps, in the order the compiler takes them.
- The lexer records trivia — comments and whitespace — rather than discarding it, so a formatter can reproduce the file exactly.
- The parser recovers from errors and emits error nodes, producing a tree that covers the whole file whether or not it is valid.
- Every node carries a span, so any consumer can map between tree positions and text offsets in both directions.
- Semantic results live in side tables keyed by node id, so a consumer that does not need types does not pay for them.
- The tree is immutable and versioned, so a background reparse can build the next version while readers use the current one.
- Diagnostics are structured values with a span, a severity and optional fixes, rather than strings printed to stderr.
- The library exposes no global state and never terminates the process on error, because it is running inside someone else's long-lived program.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- The formatter and the compiler use different parsers, and a new language feature formats to something the compiler rejects. Users learn not to run the formatter on files using that feature, and the tool loses their trust permanently.
- The parser throws on the first syntax error, so the language server has no tree for a file being edited. Completion, hover and go-to-definition all go dead the instant the user types an opening brace, and come back when they finish the block.
- The tree drops comments, and the documentation generator cannot associate a doc comment with the declaration it precedes. Docs are silently empty for the whole codebase.
- A compiler release changes a node shape, and every third-party lint rule crashes on the new version. Users cannot upgrade the compiler until the plugin ecosystem catches up, which takes months.
- Reparse is whole-file, and typing in a 5,000-line file produces visible keystroke lag. The team's conclusion is that the language is slow, and no profile ever points at the parser because each individual parse is fast.
- Two threads read a mutable tree while a background analysis mutates it. Hover occasionally returns a type belonging to a different expression, and the bug is unreproducible.
When it helps
- Any language that intends to have an ecosystem. Tooling quality is a first-order adoption factor and is downstream of this decision.
- Keeping semantics consistent across tools: one implementation of scoping means the linter, the compiler and the IDE agree about what a name refers to.
- Letting third parties build things the compiler team will never build, without them having to reimplement the language.
When it hurts
- A small internal DSL with one consumer and no external users. A library-shaped frontend with stable node APIs is a large cost for an ecosystem that will never exist — see
[[dsl-tooling-cost]]. - A frontend under heavy design churn. API stability and rapid syntax iteration are in direct conflict, and promising stability early is how a language ends up with node shapes it regrets.
- Performance-critical batch compilation, where the immutability, trivia and error-node machinery are pure overhead for a process that runs once and exits.
What it costs
Every one of these is paid by something.
- Sharing one tree buys consistency across all tools and costs the compiler team an API compatibility obligation they did not previously have, including node shapes they would rather change.
- Keeping trivia and error nodes buys formatters and editors and costs memory and node count on every parse, including the batch compilations that will never read either.
- Immutability and incrementality buy editor responsiveness and cost allocation traffic plus a strict discipline in every pass, which is enforced by convention and violated by newcomers.
- Exposing the frontend as a library buys an ecosystem and costs release velocity: a breaking change now has an unbounded blast radius outside your repository.
- Structured diagnostics buy editor integration and fixes-as-data, and cost a diagnostic system considerably more elaborate than
printf, which every message site must now use correctly.
What else you could do
What a different compiler or language does instead, and when that is better.
- Separate parsers per tool. Cheap to start, and every tool disagrees with the compiler in its own way. This was the industry default until roughly 2010 and is still common for small languages.
- A shared *grammar* rather than a shared implementation — tree-sitter's model, where editors get a fast, error-tolerant concrete tree with no semantic information at all. Excellent for highlighting, folding and structural navigation; it cannot answer a single question about types or names, so a semantic server is still required.
- A stable serialisation instead of a stable API: emit the tree as JSON or a binary format and let tools consume that. Decouples release cycles at the cost of throughput and of losing identity-based sharing entirely.
- A protocol boundary instead of a library boundary: ship the frontend as a server and expose
[[lsp]]. Tools get language support without linking anything, and you pay serialisation on every request and lose the ability to hand out the tree itself.
See it for yourself
The flag, dump or tool that shows you this directly.
- C#: the Roslyn syntax visualiser in Visual Studio shows the exact tree the compiler is using for the file you are editing, including trivia nodes and error nodes as you type.
- C / C++:
clang -Xclang -ast-dumpand libclang expose the same AST the compiler uses;clang-formatandclang-tidyare built on that library rather than on separate parsers. - TypeScript:
typescript's compiler API is the same packagetsc, the language service and every TS-aware tool use —ts.createSourceFilegives you the tree withparseDiagnosticsattached rather than thrown. - Rust: run
rust-analyzerwith logging on a file with a deliberate syntax error, and observe that hover and completion continue to work in the parts of the file that parsed. - Any editor: type an unclosed brace and watch which features survive. What still works is what was built on an error-tolerant tree; what dies was not.
Plausible wrong readings
Stated the way a confident engineer states them.
- "The compiler is the frontend's main user." It is one user, and it has the least demanding requirements of the six — it can reject bad input, run to completion, and exit.
- "A parser is a parser; any of them will do for a linter." A second parser is a second definition of the language, and the two will diverge on exactly the constructs users complain about.
- "Error recovery is a nice-to-have." It is the difference between a language server that works while you type and one that works only when your file is already correct, which is when you need it least.
- "Making the frontend a library is just refactoring." It converts internal data structures into a compatibility surface, and that is an organisational commitment, not a code change.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
Your compiler is not the only program that reads your code. The formatter, the linter, the editor, the refactoring tool and the doc generator all need to know what the program means, and if each one parses the language itself, each one is wrong differently. The fix is that they all use the same frontend, which is why compilers now ship as libraries and not only as commands.
practical
When evaluating a language's tooling maturity, ask one question: is there a single frontend library the editor, formatter and linter all use? If yes, features tend to arrive across the whole toolchain at once and tools agree with the compiler. If no, expect divergence on new syntax, formatter bugs on unusual constructs, and lint rules that flag valid code — and expect each to be fixed separately, in a different repository, on a different schedule.
advanced
The deeper consequence is that the frontend stops being a pipeline and becomes a database. A batch compiler runs phases in order and exits; a shared frontend must answer arbitrary questions about arbitrary positions at arbitrary times, with results that stay valid as the file changes underneath. That is why rustc and rust-analyzer are query engines with memoisation and dependency tracking rather than pass pipelines, why Roslyn separates immutable syntax from a workspace model that versions whole solutions, and why [[incremental-compilation]] and [[language-server]] turn out to be the same problem wearing two hats. The AST is where that shift becomes visible, because it is the first representation more than one consumer wants.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
ast module is a stable public API while the bytecode compiler behind it is explicitly not. The claim here is about tooling ecosystems, not about compiler correctness.If you were asked this in an interview
- Why does a language server not simply run the compiler on every keystroke?
- Name three requirements an editor places on a parser that a batch compiler does not.
- What does a compiler team give up by publishing its AST as a stable API?
- A formatter deletes comments in one specific construct. Where is the bug, and which design decision made it possible?
Connections
- DevOps / Production Engineering — Publishing an internal component as a versioned dependency, and the release-cadence coupling that followsTurning a frontend into a library is a dependency-management decision with the usual consequences — semantic versioning, deprecation windows, and a blast radius outside your repository. The general practice is owned there; here it is the reason a compiler team cannot change a node shape.
- Programming Languages & Runtime Internals — Long-lived process behavior: memory growth, caching and GC pressure in a server that never exitsA language server is a compiler frontend running for days. How its caches, arenas and garbage collector behave over that lifetime is a runtime concern, and it is the reason immutable trees need an allocator that expects them.