Toolingspec

Formatters

Parse the source, throw the layout away, and print it again from the tree. It only works if the tree kept the comments and blank lines the AST discards — which is why a formatter is the first tool that forces you to build a concrete syntax tree.

The question

Why does writing a code formatter require a different parser than writing a compiler?

SourceLexingTokensParsingASTSemanticsTypedIROptimizeCodegenMachine codeLinkExecute
What the program is here

A concrete syntax tree — the parse tree plus *trivia*: whitespace, blank lines, comments and the exact text of every token. The formatter reads structure from the tree and layout from nothing at all, then re-emits: every space it prints is computed from the rules, not copied from the input. The one thing it must carry across unchanged is the trivia, because comments and blank-line grouping are information the author put there and no rule can regenerate.

What this phase may assume or do

A formatting change is legal only if the emitted text parses to the same tree modulo trivia, and produces the same program behavior. That precondition is not free in every language: in JavaScript, moving a token to a new line can trigger automatic semicolon insertion and change the parse; in Python, indentation *is* syntax, so a re-indent is a semantic edit; in C and C++ a line break inside a macro definition changes the macro. A formatter must also be idempotent — format(format(x)) == format(x) — or it is not a normal form and two developers with the same tool produce oscillating diffs.

Key points

  • A formatter parses, discards the input layout, and re-emits from the tree, so the output is uniform by construction rather than by rule coverage.
  • It needs a concrete syntax tree, not an AST, because comments and blank lines are trivia the AST is designed to throw away.
  • Trivia ownership — which node a comment belongs to — is the genuinely hard part, and every formatter has an arbitrary documented rule and open bugs about it.
  • Blank lines are normalised rather than regenerated, because they carry authorial grouping no rule can reconstruct.
  • Wadler-style group-fits-or-breaks printing is why an argument list is all on one line or all broken, never partly.
  • Idempotence is required: without a fixed point, format-on-save and format-in-CI produce oscillating diffs.
  • Semantic preservation is not automatic — ASI in JavaScript, indentation in Python and macros in C all make layout semantic.
  • gofmt's no-options decision bought an ended argument, ecosystem-wide uniformity, semantic diffs, and — most underrated — codemods that produce reviewable output.

Print from the tree, not from the text

The naive idea of a formatter is a text transformer: find things that look wrong and fix them with regular expressions. That approach dies immediately, because "looks wrong" needs structure — is this brace opening a block or appearing inside a string literal? — and because the number of cases is unbounded. Every real formatter does the same thing instead: parse, discard the layout, and print the tree. The output is generated, so it is uniform by construction rather than by exhaustive rule coverage.

The classic engine is Wadler-style pretty printing, and it is worth knowing because it explains the output you see. The printer builds a document made of concatenations, nested indentation levels, and *groups*, where a group is a construct that either fits on one line or breaks all of its separators. Then it walks the document with a target width: at each group, if the whole group fits in the remaining columns, print it flat; otherwise break it. That single rule is why an argument list is either entirely on one line or entirely one-per-line, and never half and half — the behaviour people complain about is not a bug, it is the algorithm being consistent.

Consistency is the deliverable. A formatter is not trying to produce the prettiest layout for each construct; it is trying to produce *the same* layout for the same construct everywhere, so that a diff shows a change in meaning rather than a change in taste.

The group-fits-or-breaks rule, and why there is no middle
1group( width 80, fits:
2 "call(" nest(2, softline
3 a "," line b "," line c) call(a, b, c)
4 softline ")")
5
6 width 20, does not fit:
7 call(
8 aaaaaaaaaa,
9 bbbbbbbbbb,
10 cccccccccc
11 )

The group is atomic: every line inside it is a space or a newline, together. That is the whole reason you cannot get call(a, b,\n c) out of a Wadler-style printer — there is no state in which some separators broke and others did not. Formatters that appear to do otherwise are nesting several groups, so the inner one still fits while the outer one broke.

Why an AST is not enough

Here is the part that surprises people building one for the first time. An AST is designed to drop everything that does not affect meaning, and that is exactly the material a formatter must preserve. Comments do not affect meaning. Blank lines do not affect meaning. The AST throws both away — see [[parse-tree-vs-ast]] — and a formatter built on one produces beautifully formatted code with every comment deleted.

So a formatter needs a [[concrete-syntax-tree]]: every token, in order, with its surrounding trivia attached. And attaching it is where the genuine difficulty lives, because trivia ownership is ambiguous in a way structure is not. Given a comment on its own line before a function, does it belong to the function (move it when the function moves) or to whatever preceded it? Given a trailing // ok after a statement, it clearly belongs to that statement — but a comment between a closing brace and an else belongs to neither cleanly. Every formatter has a documented, arbitrary-looking rule for this, and every formatter has open issues about it.

Blank lines are the second half of the problem. They carry authorial grouping — these three statements go together — and no formatting rule can reconstruct that intent. The standard compromise, used by gofmt, Prettier, Black and rustfmt alike, is to *normalise rather than regenerate*: collapse runs of blank lines to at most one (or two at the top level), remove them at the start and end of blocks, and otherwise leave the author's choices alone. It is the one place a formatter deliberately preserves input rather than deriving output, and it is the right call.

A concrete syntax tree of x = 1; // set it, with the trivia a formatter needs
Concrete syntax tree — including trivia
ExpressionStatement“x = 1; // set it”
├── Token IDENT "x"“x”— Leading trivia: the newline and indentation before it.
├── Token EQUALS "="“=”— Trivia on both sides: one space each. The formatter regenerates these and ignores what was there.
├── Token NUMBER "1"“1”— The exact token text matters: `1`, `1.0` and `0x1` are different source and must round-trip unchanged.
└── Token SEMI ";"“;”— Trailing trivia: two spaces and `// set it`. This is what the AST would delete and what the formatter must keep.

Read it asEvery character of the input appears in exactly one node — that is the round-trip property, and it is the definition of a concrete syntax tree. The formatter regenerates the whitespace trivia from its rules and copies the comment trivia through unchanged. An AST for this statement would have two nodes, Assign(x, 1), and would be unable to tell you the comment ever existed.

gofmt has no options, and that was the point

implementationThe "no options" claim is close to true rather than exactly true, and the exceptions are instructive. gofmt has -s (simplify) and tabs-versus-spaces is fixed rather than absent; Black exposes line length and little else; Prettier shipped a dozen options and its maintainers have publicly described several as mistakes and stopped adding more; rustfmt has a large option set but marks most as unstable so the default is what almost everyone uses. The pattern across all of them is the same: the option count only ever grows, so the design decision is made at the start or not at all.

Go shipped gofmt with essentially no configuration: no indent width, no brace style, no line-length setting. This looked dogmatic in 2009 and has since been copied by Black ("the uncompromising formatter", one option that matters), by elm-format, by zig fmt and — with a handful of options that its maintainers openly regret — by Prettier. It is worth being explicit about what the decision actually bought, because "reduce bikeshedding" undersells it.

It ended the argument permanently. Not by winning it but by making it unavailable. There is no configuration file to disagree about, so no project spends time on it and no engineer carries a preference between codebases. The cost is real and should be stated plainly: your preferred style loses, and if you dislike Go's brace placement you will dislike it forever.

It made every Go file look the same, which enabled tooling. This is the underrated half. Once formatting is canonical, a tool that *rewrites* code can emit the canonical form and produce a diff containing only its own change — no formatting noise. gofix, goimports, gorename and the whole codemod tradition in Go depend on this. In an ecosystem with per-project formatting, an automated rewrite either preserves the original layout exactly (much harder) or produces an unreviewable diff.

It made diffs semantic. A diff that never contains reformatting is a diff you can review for meaning. This compounds: blame stays accurate, merge conflicts drop, and a reviewer's attention goes to the change rather than to the whitespace around it.

The generalisable lesson is not "options are bad". It is that a formatter's value comes from being a *normal form*, and every configuration option multiplies the number of normal forms in the ecosystem. One option is defensible. Forty is a style engine, and a style engine does not deliver what a formatter is for.

What the no-options decision trades awayimplementation
PropertyCanonical formatter (gofmt, Black)Configurable formatter (clang-format)
Style arguments per projectZero — there is nothing to configureRecurring, and re-litigated on every new repository
Cross-project familiarityEvery file in the ecosystem looks the sameEvery project looks different; context-switching costs
Codemod-friendlinessA rewriting tool emits canonical form; the diff is only its changeA rewriter must reproduce the project's style or produce a noisy diff
Adopting on legacy codeOne enormous reformatting commit, then never againCan be tuned to approximate the existing style and adopted gradually
Matching an existing house styleImpossible, by designThe entire point; essential where a style predates the tool
Maintenance burdenSmall: one output to test and keep stableLarge: the option matrix is the test surface, and options interact

The properties a formatter has to have

Three, and each has a failure mode you will meet if it is missing.

Idempotence. format(format(x)) must equal format(x). If it does not, the tool has no fixed point, and a repository with a format-on-save hook and a format-check in CI will oscillate: two engineers save the same file and get different bytes. This is the first property to test and the easiest to break with an ad hoc line-breaking heuristic.

Semantic preservation. The output must parse to the same tree and behave identically. Most of the time this is trivially true because only whitespace changed — but not always, and the exceptions are language-specific. JavaScript's automatic semicolon insertion means a line break before a leading ( or [ can change the parse, which is why Prettier adds a defensive semicolon in some positions. Python's indentation is syntax, so the formatter is editing the program's structure directly. C macros end at the newline. In each case, the formatter must know the language rule; a generic layout engine cannot.

Total input handling. Real files contain syntax errors while you are typing them, and an editor formats on save. A formatter that refuses to emit anything on a parse error is correct and useless in an IDE; a formatter that emits its best effort on a broken tree can destroy code. Most resolve this by declining to format when the parse fails and reporting why — which is the right default, and the reason error-tolerant parsing matters here for the same reason it does in [[language-server]].

One more thing worth stating: the formatter is not a linter, and keeping them separate is what makes both useful. A formatter produces output and no diagnostics; a linter produces diagnostics and, sometimes, fixes. When formatting rules live in the linter they compete for the same output channel as real defect findings, and the real findings lose — see [[linters]].

How it works

The steps, in the order the compiler takes them.

  • Lex the source into tokens, attaching whitespace and comments to adjacent tokens as leading and trailing trivia rather than discarding them.
  • Parse into a concrete syntax tree in which every byte of the input appears exactly once, so that printing the tree unchanged reproduces the file.
  • Walk the tree and build a layout document: text, nested indentation, groups, and separators that are either a space or a newline depending on whether their group breaks.
  • Copy comment trivia into the document at the position dictated by the ownership rules, and normalise whitespace trivia away entirely.
  • Render the document against a target width, deciding per group whether it fits flat and breaking every separator in the group if it does not.
  • Handle constructs the layout engine cannot decide — a comment that forces a break, a preserved blank line, a language construct whose formatting is fixed — with explicit hard breaks.
  • Verify before writing: re-parse the output and compare against the input tree modulo trivia, which is the check that turns a formatting bug into a refusal instead of a corrupted file.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • The formatter deletes a comment, or moves it from the line it documented to the line above, and the loss is invisible in a large reformatting diff.
  • Running the formatter twice produces different output, so a save hook and a CI check disagree and a file changes on every commit regardless of what the author edited.
  • Formatting introduces or removes an automatic semicolon in JavaScript and the program's behavior changes silently — the reason a formatter must know the language rather than just the grammar.
  • A first-time adoption produces a repository-wide reformat that destroys git blame for every line and makes every in-flight branch conflict.
  • The formatter is not idempotent on one specific construct, and a file oscillates between two forms as different developers save it.
  • A file with a syntax error is formatted "best effort", the broken tree prints wrong, and code is lost — which is why the correct behaviour is to refuse and say why.

When it helps

  • Any repository with more than one contributor, where the value is the elimination of an entire category of review comment rather than the aesthetics.
  • Codemods and automated rewrites, which can emit canonical output and produce a diff containing only the semantic change.
  • Code review, where a diff guaranteed free of reformatting is a diff you can read for meaning.
  • Generated code, which is far more reviewable when it goes through the same formatter as hand-written code.

When it hurts

  • Retrofitting onto a large repository with active branches: the one-time reformat conflicts with everything in flight and rewrites blame. The mitigation is to land it as a single commit and add it to .git-blame-ignore-revs, and to time it deliberately.
  • In languages where layout is load-bearing beyond indentation — assembly with column-aligned operands, embedded DSLs inside string literals, or hand-tuned tables where alignment is the documentation.
  • When the formatter and a linter disagree, which produces an infinite loop between two tools. The fix is always to remove the formatting rules from the linter, not to configure both.
  • For a codebase with a strong pre-existing house style that predates the tool, where a canonical formatter means rewriting everything and a configurable one means the maintenance burden of an option matrix.

What it costs

Every one of these is paid by something.

  • A canonical formatter with no options buys an ended argument and ecosystem uniformity, and pays by making your preferred style permanently unavailable and by forcing a one-time repository-wide rewrite.
  • A configurable formatter buys the ability to match an existing house style and adopt gradually, and pays a growing option matrix that becomes the test surface and a per-project configuration to maintain forever.
  • Requiring a concrete syntax tree buys comment and blank-line fidelity and pays a second tree representation to build, test and keep in sync with the compiler's AST.
  • Formatting on save buys never thinking about layout and pays an editor round-trip on every save plus the risk that a non-idempotent formatter fights the CI check.
  • Verifying the output by re-parsing buys protection against corrupting a file and pays roughly a second parse on every format, which is why some tools do it only in a debug mode.
  • Enforcing formatting in CI buys a guarantee that the repository is canonical and pays a failed build for a whitespace difference, which is friction unless the fix is a one-line command.

What else you could do

What a different compiler or language does instead, and when that is better.

  • An editor-side style configuration (EditorConfig, per-editor settings) applies to what you type rather than rewriting the file, which avoids the big-bang reformat and guarantees nothing about what lands.
  • Formatting rules in a linter with autofixes, which is what JavaScript did before Prettier. It works, and it competes with real findings for the same output channel — see [[linters]].
  • A structural editor or projectional editor, where the source is stored as a tree and layout is purely a rendering decision, so formatting stops existing as a problem. Genuinely solves it, and costs the ability to use ordinary text tools.
  • No formatter and a style guide enforced by review, which is the historical default. It costs reviewer attention indefinitely and delivers less uniformity than any tool.
  • A canonical *diff* rather than canonical source: tools that ignore whitespace when comparing (git diff -w, semantic diff tools) address the review symptom without addressing the storage.

See it for yourself

The flag, dump or tool that shows you this directly.

  • Go: gofmt -d file.go prints the diff without writing; gofmt -l ./... lists non-conforming files, which is the CI form; gofmt -s also applies simplifications. go fmt ./... is the wrapper.
  • Rust: cargo fmt -- --check for CI, rustfmt --print-config default rustfmt.toml to see every option and its default, and --emit=stdout to preview.
  • JavaScript/TypeScript: npx prettier --check . in CI, npx prettier --write ., and npx prettier --debug-print-doc file.ts — the last one prints the intermediate layout document, which is the single best way to understand why a construct broke where it did.
  • Python: black --diff --check file.py, and black --line-length 100 for the one option that matters.
  • C/C++: clang-format --dump-config -style=llvm prints the full option set as YAML, clang-format -n --Werror checks without writing, and -style=file uses the project .clang-format. git clang-format formats only the staged diff, which is how to adopt without a big-bang reformat.
  • Any repository: after a one-time reformat, record the commit in .git-blame-ignore-revs and set git config blame.ignoreRevsFile .git-blame-ignore-revs, or blame is degraded for everyone permanently.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "A formatter is a cosmetic tool." Its real product is a normal form, and the normal form is what makes diffs semantic and codemods reviewable. The aesthetics are incidental.
  • "I can write one with regular expressions." You can write one that works on the examples you tried. Structure is required to know whether a brace is code or text, and comments are required to be preserved, and neither survives a regex.
  • "An AST is enough — formatting does not change meaning." Formatting does not change meaning, which is exactly why the AST discarded the material the formatter needs. Comments and blank lines are not in the AST.
  • "The formatter chose a strange line break." Almost always the group did not fit, so every separator in it broke. That is one rule applied consistently, not a heuristic misfiring.
  • "Formatting is safe by definition." Not in JavaScript, where a line break can trigger semicolon insertion; not in Python, where indentation is syntax; not in C, where a macro ends at the newline.

Misconceptions

The claim, and what is actually true.

A formatter and a linter do the same kind of work.
A formatter emits a normal form and produces no diagnostics; a linter produces diagnostics and sometimes fixes. Merging them puts style complaints in the channel where real defect findings appear.
Configurable is strictly better than canonical.
Every option multiplies the number of normal forms in the ecosystem, which is precisely the property that made codemods and noise-free diffs possible in Go. Configurability buys gradual adoption and sells uniformity.
Formatting the whole repository is a harmless commit.
It rewrites blame for every line and conflicts with every open branch. Doable, and it needs .git-blame-ignore-revs and deliberate timing.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

A formatter reads your code into a tree, forgets how it was laid out, and prints it again by its own rules — which is why the result is consistent everywhere instead of consistent only where somebody remembered. The catch is that the tree a compiler builds deliberately drops comments and blank lines, because those do not affect meaning. A formatter needs them, so it needs a richer tree that keeps every character of the original. That single requirement is why formatting tools are built on different parsers than compilers.

practical

Adopting one: pick the canonical tool for your language, run it over everything in one commit, add that commit to .git-blame-ignore-revs, and add --check to CI. Then delete every formatting rule from your linter — if you skip this the two tools will fight and the loop is genuinely infinite. Pin the formatter version, because an upstream release will otherwise reformat files nobody touched and turn CI red. And when the output surprises you, run the debug printer (prettier --debug-print-doc, or read the group structure) rather than adding an override; the answer is almost always that a group did not fit and every separator inside it broke together.

advanced

The engineering core is the layout document plus a fits-in-remaining-width decision per group, and the interesting extensions are all about escaping its uniformity where uniformity is wrong. Prettier and rustfmt both preserve the author's choice for certain constructs — an object literal that had a newline after the brace stays expanded — because the algorithm alone produces worse results than the author's intent for data-shaped code. That is a deliberate retreat from "output is a pure function of the tree" to "output is a function of the tree plus a small amount of preserved input", and it is the same compromise as blank lines. The lesson generalises: a normal form is valuable in proportion to how few exceptions it has, and every exception you add is bought from the property that made the tool worth having. Knowing which exceptions a formatter made — and being able to name them — is what separates configuring one from understanding it.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

specWhether formatting can change program behavior is a language-specification question, not a tool-quality one. JavaScript's automatic semicolon insertion is defined in ECMA-262 and makes line breaks semantically significant in specific positions; Python's grammar makes INDENT and DEDENT real tokens; the C preprocessor defines a directive as ending at a newline. In Go, Rust and Java, by contrast, layout carries no meaning outside string literals, which is exactly why formatters for those languages are simpler and safer.
implementationOption counts and defaults move between releases. Prettier has added and then frozen options and documents several as regretted; rustfmt gates most of its options as nightly-only so the stable default is near-canonical; clang-format's option set has grown steadily and its bundled styles (llvm, google, chromium, mozilla, webkit) are themselves moving targets. Pin the formatter version in CI, or an upstream release reformats files nobody edited.
typicalThe blank-line rule described here — collapse runs, strip at block boundaries, otherwise preserve — is what gofmt, Black, Prettier and rustfmt converge on, but the exact limits differ (one blank line inside a function versus two at the top level, and different treatment around comments). It is a convergent convention rather than a requirement, and a formatter is free to regenerate blank lines entirely; none of the popular ones does, because the result reads worse.

If you were asked this in an interview

  • Why does a formatter need a concrete syntax tree rather than an AST?
  • What does idempotence mean for a formatter and what breaks without it?
  • gofmt has no options. Argue both sides of that decision.

Connections

Domains that do not exist yet
  • DevOps / Production Engineering — Landing a repository-wide reformat without destroying blame or blocking every open branch
    The technical part of adopting a formatter is one command; the hard part is sequencing a commit that touches every file against in-flight work, .git-blame-ignore-revs, and a CI check that must not become friction. That sequencing is a delivery-engineering problem owned there, and it is the reason most adoptions stall.