AtlasLang: What a Language Owes Its Users
A working compiler is the smaller half. Once people write programs in your language they need diagnostics that point at the mistake, a formatter that ends the argument, highlighting that is right about their code, and eventually a language server — because everything they use is going to want one.
The compiler works. What else does a language need before anyone can use it?
The frontend as a queryable service rather than a batch process: the same lexer, parser and checker, but exposed so that a caller can ask "what is at this position", "what does this name refer to", "what is wrong with this file" and get an answer in milliseconds against source that is currently being edited and is frequently invalid. The compiler already computes all of it; the difference is the shape of the interface and the tolerance for broken input.
A tool built on the frontend may assume only what the frontend guarantees on *invalid* input, which is a much weaker set than a batch compiler needs. It gets: a token list with spans that slice back to their text, an AST that exists even after parse errors thanks to recovery and placeholder nodes, and diagnostics that do not cascade because the error type is absorbing. It does not get a complete or correct program. Any tool that requires one — a formatter that reprints from a valid AST, a refactoring that assumes resolution succeeded — must detect that assumption failing and decline, because the alternative is a tool that silently mangles a file its user was midway through typing.
Key points
- A working compiler is the smaller half of shipping a language; diagnostics, formatting, highlighting and editor integration are what people actually touch.
- A good diagnostic carries a span, names what was found as well as what was expected, and uses the help line to teach the rule that was broken.
- Non-cascading errors matter more than message wording: one missing brace should produce one message, not twenty.
- Suggestions need a distance budget, because a confident wrong suggestion that compiles is worse than silence.
- A formatter's value is social — it ends an argument permanently — and its technical precondition is a tree that keeps comments, which an AST does not.
- Semantic highlighting from the compiler is exact where a regex grammar guesses, and its cost is that the frontend must keep up with typing.
- A language server is not a feature but four requirements: tolerate broken code, answer in milliseconds, keep what a batch compiler discards, and answer positional questions.
- AtlasLang meets two of those four by design and fails one by discarding comments, which is why it has no formatter.
Diagnostics are the interface most people meet first
A compiler's error messages are, for most users, the largest part of the language they interact with. AtlasLang's carry three things deliberately, and the third is the one usually skipped.
A span, not a line number. Every diagnostic covers a byte range, so an editor can underline exactly the offending text. That range came from the lexer and survived through the parser because AST nodes take their spans from the tokens they were built from — which is why [[atlaslang-lexer]] insisted on it.
What was found, not only what was expected. "Expected ; after a let binding" with the help "Found an identifier instead" tells the user both halves. Either alone leaves them guessing: expected-only makes them hunt for what is actually there, found-only makes them guess what would have been acceptable.
The rule, in the help line. "An if condition must be bool, found int" is the error; "AtlasLang has no truthiness; write an explicit comparison" is the help. The second sentence teaches a language rule to someone who did not know it, and the help line is the only place in the whole system where that teaching can happen at the moment it is needed.
Two mechanisms keep the output readable rather than merely correct. The parser marks diagnostics raised before it has resynchronized as cascading, so one missing brace does not bury the real error under its consequences. And the checker's error type is absorbing, so one undefined name produces one message rather than one per enclosing expression. Both are small, and both matter more than the wording of any individual message — see [[diagnostic-quality]].
Suggestions come with a budget. An unresolved name triggers an edit-distance search over visible names, and the suggestion is offered only within one edit for short names or a third of the length for longer ones. Past that it stays silent, because a confidently wrong suggestion that happens to compile is worse than none — [[suggested-fixes]].
A formatter is a social artifact
The technical description of a formatter is dull: parse, then print from the tree by fixed rules. The reason to ship one is not technical at all. A formatter with no options ends every formatting argument in every code review for the life of the language, permanently, and that is worth more than any individual formatting decision it makes. Go proved this and Rust, Python and JavaScript all followed with varying degrees of optionality — and the ones with more options got proportionally less of the benefit.
The technical problem it creates is real though: printing from an AST loses everything the AST does not keep. AtlasLang's parser discards comments and whitespace entirely, and drops the parentheses in (a + b) * c because the tree records the grouping structurally. A formatter built on this AST would delete every comment in the file and would have to re-derive parentheses from precedence. Deleting comments is not a bug you ship twice.
The fix is a different tree. A concrete syntax tree keeps every token, including whitespace and comments, attached as trivia — so a formatter can reprint faithfully and a refactoring tool can rewrite one expression while leaving the rest of the file byte-identical. That is why Roslyn, rust-analyzer and every serious modern frontend build one, and it is a decision that has to be made early, because retrofitting trivia into a parser means touching every node constructor. AtlasLang chose not to, which is why it has no formatter and why [[concrete-syntax-tree]] and [[formatters]] are lessons about a road not taken here.
Highlighting, and the honest version of it
Syntax highlighting looks like the easiest tool to build and is where most languages first lie to their users. The regular-expression version — the one every editor ships as a language grammar file — colours let because it matches a keyword pattern, and colours the word let inside a string comment the same way if the pattern is sloppy. It cannot tell a type name from a variable that happens to share it, cannot tell a shadowed variable from the one it shadows, and cannot know that a call is to a function that does not exist.
The honest version colours from the compiler. The lexer already classified every token, so keyword-versus-identifier is free and exact — there is no way for a keyword inside a string to be mis-coloured, because the string is one token. The checker already resolved every identifier, so a name can be coloured by what it *is*: a parameter differently from a local, a function differently from a variable, an unresolved name differently from a resolved one. That is semantic highlighting, and once you have seen a shadowed variable coloured distinctly from its outer namesake it is difficult to go back.
The cost is latency. A regex grammar highlights instantly and is never wrong about being fast; a semantic highlighter has to have parsed and checked the file, which means highlighting either lags editing or requires the frontend to be fast enough to keep up with typing. That requirement — the frontend re-running on every keystroke — is what actually drives modern compiler architecture, and it is the subject of the next section.
The language server, and what it demands of a compiler
Every feature above converges on one component. A language server is a process holding the frontend, receiving edits, and answering questions — diagnostics, hover, go-to-definition, completion, rename, find-references — over a protocol that any editor can speak. Writing one server rather than one plugin per editor is the whole point of the Language Server Protocol, and it is why a new language today gets editor support in every major editor or in none.
What is worth understanding is that a language server is not a feature added to a compiler. It is a set of requirements that changes how the compiler is built, and there are four of them.
It must tolerate broken code. A batch compiler sees a file its author considered finished; a language server sees a file mid-keystroke, which is invalid most of the time. Every feature has to work on a file with a syntax error in it, which means error recovery and placeholder nodes are not a nicety but the foundation. AtlasLang's panic-mode recovery and absorbing error type are exactly what this needs.
It must be fast enough to be interactive. Completion after a hundred milliseconds is useful; after a second it is ignored. On a large project you cannot recompile everything per keystroke, which forces a demand-driven, memoised architecture where a query recomputes only what a change actually reached — the same machinery as [[incremental-compilation]], driven by a much harsher deadline.
It must keep what a batch compiler discards. Comments, for hover documentation. Whitespace, for formatting and for rename. Every span, for every navigation feature. The loses column from [[atlaslang-overview]] is a list of things a language server needs and a batch compiler may throw away — which is why the two are increasingly the same codebase, and why the one that was written batch-first usually has to be rebuilt.
It must answer positional questions. "What is at byte 412" is not a question a batch compiler ever asks, and answering it efficiently needs the spans threaded through every representation rather than kept only where a diagnostic might need them.
AtlasLang meets the first, has no need of the second at its scale, fails the third by discarding comments, and meets the fourth because spans are threaded everywhere. That is a fair scorecard for a teaching implementation and an honest one: it is why the pipeline explorer can highlight across panels, and why there is no formatter.
| Tool | Needs | Does AtlasLang have it? |
|---|---|---|
| Diagnostics with underlines | Spans on every token and node | Yes — threaded through every stage |
| Multiple errors per compile | Error recovery and non-cascading diagnostics | Yes — panic-mode recovery and an absorbing error type |
| Semantic highlighting | Resolved symbols on identifiers | Yes — the checker writes a symbol on every identifier |
| Go-to-definition | A symbol table with declaration positions | Yes — SymbolInfo records declaredAt |
| Hover with documentation | Comments retained and attached to declarations | No — comments are discarded by the lexer |
| A faithful formatter | A tree that keeps trivia and original token text | No — the AST drops whitespace, comments and parentheses |
| Rename across a file | Symbol identity plus every use site's span | Yes for identity; the use sites are enumerable from the tree |
| Sub-second response on a large project | Demand-driven recomputation with memoised queries | Not needed at this scale, and not present |
How it works
The steps, in the order the compiler takes them.
- Every diagnostic carries a byte range that originated in the lexer and survived the parser because AST nodes inherit their spans from their tokens.
- The parser recovers from a syntax error by synchronizing to a statement boundary and marks anything reported before resynchronizing as cascading.
- The checker gives a failed node an absorbing
errortype, so one mistake produces one message rather than one per enclosing expression. - An unresolved name triggers an edit-distance search over visible names, suggesting only within a length-scaled budget.
- Every identifier receives a symbol id, and every symbol records the offset it was declared at — which together are go-to-definition and semantic highlighting.
- A language server wraps the same frontend behind a request loop, holding the current text and re-running the queries an edit invalidated.
- A formatter would parse and reprint by fixed rules, which requires a tree retaining comments and whitespace as trivia — a different tree from this AST.
- An incremental architecture memoises queries and records their dependencies so an edit recomputes only what it reached, which is what makes millisecond responses possible on a large project.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- A single missing brace produces twenty confident diagnostics and the real one is third from the bottom, so users learn to read only the first message and miss the ones that matter.
- A "did you mean" suggestion for an unrelated name is accepted, the program compiles, and it does the wrong thing — worse than no suggestion.
- A formatter built on a comment-discarding AST reformats a file and deletes every comment in it, which is discovered after it has been run across a repository.
- A regex-based highlighter colours a keyword inside a string, or colours a shadowed variable identically to the one it shadows, and users trust the colours anyway.
- A language server built on a batch compiler recompiles the world per keystroke, completion arrives after two seconds, and users disable it.
- A frontend that throws on the first parse error makes every editor feature stop working while the file is mid-edit, which is most of the time.
- A refactoring tool that assumes name resolution succeeded silently mangles a file with an unresolved name in it, instead of declining.
When it helps
- Deciding whether to build a language at all, where this is most of the true cost and it is the part usually left out of the estimate —
[[dsl-tooling-cost]]. - Prioritising work on an existing language, where non-cascading diagnostics and error recovery return more per hour than almost anything else.
- Designing a frontend that will eventually need a language server, where retaining trivia and threading spans are decisions that are cheap now and expensive later.
- Evaluating a language's maturity honestly: the compiler working says less than whether an editor can rename a symbol in a file with a syntax error in it.
When it hurts
- For an internal DSL with three users, where a language server is a year of work for an audience who would have been fine with a config file.
- As a checklist to complete before shipping. Diagnostics and error recovery come first by a wide margin; the rest can follow the users who ask for it.
- When tooling ambition drives the language design — a feature dropped because it would be hard to complete on is sometimes right and often the tail wagging the dog.
What it costs
Every one of these is paid by something.
- Retaining trivia buys a formatter and faithful refactoring and pays with a larger tree, a more complex parser, and every node constructor needing to carry it.
- Semantic highlighting buys exactness and pays in latency, because the frontend must have parsed and checked before anything can be coloured.
- A demand-driven incremental architecture buys interactive response on a large project and pays in substantial implementation complexity, bookkeeping memory, and usually a slower cold build.
- Error recovery buys multiple diagnostics per compile and every editor feature working on broken code, and pays with cascading messages that must be detected and demoted.
- A formatter with no options buys the end of an argument forever and pays by imposing a decision on people who disagree with it, some of whom will never forgive it.
- Shipping a language server buys editor support everywhere at once and pays with a component that must be maintained against a protocol you do not control.
What else you could do
What a different compiler or language does instead, and when that is better.
- Ship no tooling and rely on generic text-editor support, which is honest for a small internal language and is what most DSLs actually do.
- Generate an editor grammar file — TextMate or Tree-sitter — which gives fast, approximate highlighting for a fraction of the effort and is genuinely the right first step.
- Embed the language in a host language as an internal DSL, inheriting all of the host's tooling for free — the strongest argument in
[[internal-vs-external-dsl]]. - Reuse an existing frontend framework rather than building from scratch, which is what Tree-sitter for parsing and the LSP libraries for the protocol both offer.
- Compile to another language and let its tooling handle the rest, which is what many small languages do and which trades tooling cost for debugging that points at generated code.
See it for yourself
The flag, dump or tool that shows you this directly.
- Load the
errorsexample on/compilers/atlaslang: a lexer error, a parse error and a type error in one file, each with a span and a help line, with recovery between them. - Type
let a = 1with no semicolon followed by more statements, and see which diagnostics are marked cascading. - Type an undefined name close to a defined one and see the suggestion; then move it further away and watch the suggestion stop.
/compilers/pipeline— clicking a node highlights its origin in every other panel, which is the same span machinery a language server's go-to-definition uses.- Compare with a real one: run
rust-analyzerorgoplswith logging on and watch the request traffic while typing — the volume is what forces the incremental architecture. src/compilers/sim/check.ts—SymbolInfocarriesdeclaredAtandannotated, which are go-to-definition and "was this inferred" respectively.
Plausible wrong readings
Stated the way a confident engineer states them.
- "Tooling comes after the language is finished." The tooling requirements change how the frontend is built, and retrofitting trivia or incrementality into a batch compiler usually means rewriting it.
- "A language server is a plugin." It is a process holding the compiler frontend, subject to four requirements a batch compiler does not have — chiefly that it works on code that does not compile.
- "Syntax highlighting is easy." Approximate highlighting is easy. Highlighting that is right about shadowing, unresolved names and the difference between a type and a variable requires the checker.
- "Good diagnostics are a matter of writing better messages." They are mostly a matter of recovery, non-cascading errors and spans. The wording is the last five percent.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
Once people use your language they need more than a compiler: error messages that point at the right place and explain the rule, a formatter so nobody argues about layout, highlighting that is actually right, and editor support for jumping to a definition or renaming a variable. Most of that is built from things the compiler already computes — the spans, the symbol table, the types — exposed differently.
practical
Do them in this order. Spans on everything, from day one, because they cannot be added back cheaply. Then error recovery and non-cascading diagnostics, which are worth more than any other tooling work per hour spent. Then a Tree-sitter or TextMate grammar for approximate highlighting, which is a day's work and covers most editors. Then, if the language is going anywhere, decide about trivia in the parser *before* you have a hundred node types, because that decision is nearly free now and a rewrite later.
advanced
The deepest consequence of tooling is that it reshapes the compiler. A batch compiler is a pipeline that runs once over a valid program and is free to discard whatever the next stage does not need. A language server needs the same frontend to run continuously over an invalid program, answer positional queries in milliseconds, and retain everything — which is why modern frontends are demand-driven query engines rather than pipelines, and why rust-analyzer and rustc converged rather than staying separate. The general principle is that the interactive requirement is strictly stronger than the batch one: anything that can answer a question about a half-written file in ten milliseconds can also compile a finished one, and nothing built for the second case is likely to satisfy the first. A language designed today that expects to be used seriously should assume its frontend will eventually be an IDE, and pay the architecture cost while the codebase is small enough that it is a choice rather than a rewrite.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
If you were asked this in an interview
- What does a language server require from a compiler frontend that a batch compiler does not?
- Why can a formatter not be built on this AST, and what would have to change?
- You have one week to improve a young language's usability. What do you do, and why not the formatter?
Connections
- Developer Experience and Documentation — What people need around a language beyond its tooling: a tutorial, a reference, examples, and a package ecosystemDiagnostics and editor support are the parts of developer experience the compiler can supply. The rest — teaching material, discoverability, a way to share code — decides whether a language is adoptable at all, and none of it is a compiler concern.