Recovering Structure From Model Output
The practical lesson: how to get a reliable data structure out of text a model wrote. A strict parser with real error recovery beats a pile of regular expressions for the same reasons it does in a compiler, repair-and-retry is a legitimate strategy with a cost worth naming, and no parser will ever tell you whether the output meant what the user wanted.
The model wrapped its JSON in prose and left a trailing comma. Do I regex it, repair it, or reject it?
A byte string with no guaranteed structure, and the sequence of forms you are trying to move it through: characters, tokens with spans, a syntax tree, and finally a typed record. Each step is a real phase with its own failure class — a lexical error is a bad character, a syntax error is a well-formed token in the wrong place, and a type error is a well-placed value of the wrong kind. Collapsing them into "parsing failed" throws away exactly the information that would tell you what to do next.
A repair may be applied only when the grammar makes the edit unique at that position — the parser knows what it expected, exactly one minimal edit produces a parse, and the edit cannot change any value the input already committed to. Deleting a trailing comma qualifies; inventing a digit to complete a truncated number does not, because the input did not commit to a value and the repair would be choosing one. Where the edit is not unique, the correct action is to reject with a diagnostic and let a retry or a human decide, since a plausible wrong repair is strictly worse than a rejection: it executes.
Key points
- Regular expressions cannot find the extent of a nested structure; that is a property of regular languages, not a gap in your pattern.
- A parser gives you a byte offset, what it expected and what it found — which is what makes a repair prompt precise instead of a re-roll.
- Recover and keep parsing so one round trip reports every problem, and suppress cascading diagnostics so the first real error is visible.
- Apply a repair only when the grammar makes the edit unique and the edit changes no value the input committed to.
- Never repair a truncation. Completing a truncated value means choosing a value, and the value is the thing that matters.
- Distinguish the failure class before retrying: retrying a parse failure is sensible, retrying a policy denial burns the whole budget.
- A parser proves syntax and a checker proves kinds. Neither says anything about intent, and past a point the marginal engineering belongs elsewhere.
Why a parser, when a regex is three lines
The regex approach works until the model writes a JSON object containing a string that contains a brace. Then the greedy match runs past the end, the lazy match stops early, and the failure is silent — you get a truncated object that happens to parse, or a parse error whose position means nothing. Every fix adds a case, and the cases interact.
This is the same argument the domain makes about lexing and parsing generally: a regular language cannot describe balanced delimiters, so no amount of regular-expression cleverness will correctly find the extent of a nested structure. [[regular-languages]] is the theory; [[what-parsing-does]] is the practice. A parser tracks nesting depth and string state because those are exactly the things a regular language cannot express.
The second reason is diagnostics. A regex that fails to match tells you nothing. A parser that fails tells you the byte offset, what it expected there, and what it found instead — which is the difference between a repair prompt that works and a re-roll that hopes. That precision is bought entirely by having recorded spans, which is why [[source-locations]] is a lesson and not a footnote.
{"tool": "refund", "amount": 20}Read it asNine tokens with byte ranges. The whitespace at offsets 8, 18 and 28 produced no token and is unrecoverable from the stream alone — which is fine here and would not be if you wanted to echo the model's formatting back. Note what the tokens still do not tell you: that refund is a tool that exists, whether amount is dollars or cents, or whether this caller may issue refunds. Those are three later phases, and [[typed-tool-calls]] is the next one.
Error recovery: keep parsing, report everything
A parser that stops at the first error makes the caller fix one problem per round trip, and each round trip is a model call. A recovering parser reports the trailing comma, the missing closing brace and the unquoted key in one pass, and one repair prompt fixes all three.
The technique is the same one AtlasLang's own parser uses: panic-mode recovery. On an error, record it, then skip tokens until reaching one that plausibly starts the next construct — for JSON, a comma at the current nesting depth or the closing brace — and resume. [[error-recovery]] and [[parser-synchronization]] cover the general form, and the AtlasLang implementation in [[atlaslang-parser]] synchronizes on ; and the statement keywords for exactly the same reason.
Recovery also needs cascade suppression. Once the parser is confused, the next three errors are usually artefacts of the first, and reporting them all buries the real one. AtlasLang flags cascading diagnostics rather than hiding them; a repair prompt should carry the first error and the count, not all of them.
{"tool": "refund", "amount": 20,}{"tool": "refund", "amount": 20}The parser was at a position where the grammar admits either another member or }. Exactly one minimal edit — deleting the comma — yields a parse, and the edit removes a separator rather than supplying a value, so no value the input committed to is changed. Under those two conditions the repair is determined by the grammar rather than chosen by the repairer, which is what makes it safe to apply without asking anyone.
The output is truncated mid-value: {"tool": "refund", "amount": 2. Completing it requires inventing the rest of a number, and the grammar admits 2, 20, 200 and 2000 equally. Deleting a separator cannot change what gets refunded; supplying a digit can turn a two-cent refund into a twenty-dollar one. Truncation must be retried or rejected, never repaired — and the same rule rules out "helpfully" adding a missing closing brace when the content before it may itself be incomplete.
Repair, retry, or reject
These are three different strategies with three different costs, and the mistake is treating them as one fallback chain applied uniformly. Deterministic repair is free and should be applied wherever the edit is grammar-determined. Retry costs a full model call and, importantly, produces a *different* output — so a bug that manifested once may not reproduce, and your success metric now includes attempts nobody sees.
Rejection is the one people skip, and it is the correct answer more often than it feels. If the output is truncated, if two repairs are equally plausible, or if the same field has failed twice, the information you have does not support a decision, and the honest move is a structured refusal that says which field and why.
The cost that hides best is the retry budget interacting with the failure class. Retrying a parse failure is reasonable — the sampler may do better. Retrying a *policy* failure is not, and a repair loop that does not distinguish them will spend the whole budget watching a model rewrite the shape of an action it is never going to be allowed to take. That is the [[plan-validation]] gate distinction paying for itself.
| Strategy | Costs | Correct when |
|---|---|---|
| Deterministic repair | Nothing at run time; a repair table to maintain and test | The grammar makes the edit unique and it changes no committed value |
| Constrained re-decode | A model call; provider support; some quality on hard cases | The failure is syntactic and the provider supports grammar constraints |
| Repair prompt and retry | A full model call, added latency, and a different output each time | The diagnostic names a specific field and the failure class is generation, not policy |
| Reject with a diagnostic | Task completion; a user-visible failure | Truncated output, ambiguous repair, or the same field failing twice |
| Regex extraction | Correctness — it cannot find the extent of a nested structure | Genuinely never, for structured output; use a parser |
The honest limit
A parser proves syntax. A type checker proves that values are of the declared kinds. Together they establish that the output is a well-formed instance of the thing you asked for — and that is the entire extent of it. Whether the model understood the request is not a property of the text, and no amount of grammar work will make it one.
It is worth being blunt about this because the effort curve is deceptive. Going from regex to parser is a large, real gain. Going from a good parser to an excellent one gains very little, because the remaining failures are semantic and live entirely outside what any parser can see. Past a certain point the marginal engineering belongs in evaluation, tool design and review, not in the frontend.
The compiler analogy holds all the way down, which is the reassuring part. A C compiler that accepts your program has told you it is well-formed. Whether it computes what you meant is between you and your tests — and nobody has ever expected otherwise from a compiler.
How it works
The steps, in the order the compiler takes them.
- Tokenize the output, recording a byte range on every token, so every later diagnostic can quote the exact substring that caused it.
- Parse against the format's grammar, tracking nesting depth and string state — the two things a regular expression structurally cannot track.
- On an error, record message, span, and what was expected; then synchronize to the next token that can start a construct at the current depth and continue.
- Flag diagnostics reported while still recovering as cascading, so the caller can show the first and count the rest.
- Apply only grammar-determined repairs from an explicit table, and record which repairs fired so the rate is visible.
- Classify the failure — lexical, syntactic, type, policy — and choose repair, re-decode, retry or reject from that class rather than from a single fallback chain.
- When retrying, include the specific diagnostic and span in the repair prompt, and cap retries per field rather than per request.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- A greedy regex swallows a brace inside a string, the extracted text still parses, and a truncated object is executed as though it were complete.
- The system silently retries three times on every request and nobody notices, because only the final success is logged; latency is triple what anyone believes and the cost is triple too.
- An auto-repair completes a truncated number, and a refund is issued for the wrong amount with no error anywhere in the trace.
- The parser stops at the first error, so a repair prompt fixes one problem and reveals the next, and a plan with three formatting mistakes costs four model calls.
- Cascading diagnostics are all reported, the repair prompt contains eleven errors of which one is real, and the model rewrites the whole output rather than fixing the actual problem.
- A policy denial is fed into the format-repair path, and the retry loop rewrites the same forbidden action in new shapes until the budget is gone.
- Everything parses, everything types, and the agent confidently does the wrong thing — and the team responds by strengthening the parser, which cannot help.
When it helps
- Any provider or model that does not support constrained decoding, where malformed output is a fact of life rather than an edge case.
- Formats richer than a single JSON object — a document with sections, a plan with embedded code, a mixed response — where the structure has to be recovered rather than assumed.
- Debugging generation regressions, since a distribution of parse-failure positions points at the exact construct the model has started getting wrong.
- Reducing cost, because a deterministic repair that fires on 3% of outputs removes 3% of your retry traffic entirely.
When it hurts
- When constrained decoding is available and sufficient: building a repair layer for output that cannot be malformed is maintaining a table that never fires.
- When the repair table grows past a handful of grammar-determined edits and starts encoding guesses; at that point it has become a source of silent wrong answers.
- When the real problem is semantic. A stronger parser cannot fix a model that misunderstood the request, and time spent there is time not spent on evals.
What it costs
Every one of these is paid by something.
- A real parser buys correct extents and precise diagnostics, and costs a grammar to maintain and a genuine dependency where three lines of regex used to sit.
- Error recovery buys one round trip instead of several, and costs cascade-suppression logic plus the risk that a recovered parse produces a tree confident enough to execute and wrong.
- Deterministic repair buys latency and retry traffic, and costs a table that must stay provably grammar-determined — the first "helpful" entry that guesses a value converts a rejection into a silent wrong answer.
- Retry buys completion rate and costs latency, tokens and reproducibility: the retried output is a different output, so a bug that appeared once may never appear again in the same form.
What else you could do
What a different compiler or language does instead, and when that is better.
- Constrained decoding, so malformed output is unsamplable — the strongest option where the provider supports it, and it is
[[typed-tool-calls]]'s subject rather than this one's. - A line-oriented or delimiter-heavy format instead of nested JSON, so a partial output is still usable up to the last complete record. Much more robust to truncation, and unable to express nesting.
- Two-stage generation: a small, heavily constrained call for the structure and a free-form call for the prose, so the part that must parse is short. More calls, far fewer parse failures.
- Accepting free-form output and having a second model extract the structure. Sometimes the only option for messy inputs, and it replaces a deterministic parser with a second probabilistic component, which is a real downgrade in debuggability.
See it for yourself
The flag, dump or tool that shows you this directly.
- Log the raw output on every parse failure, not just the error. Without the input, a parse-failure rate is a number with no diagnosis attached.
- Chart parse failures by byte position relative to output length. A cluster at the very end is truncation — a token-limit problem, not a format problem.
- Count repairs by rule. A rule that fires often is telling you about a systematic model behaviour that a prompt or schema change could remove at the source.
- Log retry counts per request and look at the distribution, not the mean. The tail is where the latency complaints come from.
- Run the AtlasLang error example at
/compilers/atlaslang— pastelet n = 123abc;followed bylet ok = 1 +;andlet bad: bool = 5;— and watch a lexical, a syntactic and a type error each get reported once, with spans, from a single pass. That is what recovery plus cascade suppression looks like when it works.
Plausible wrong readings
Stated the way a confident engineer states them.
- "JSON.parse in a try/catch is a parser." It is a parser with one bit of output. It cannot tell you where the problem was, what was expected, or whether it was recoverable, and those are the three things you need.
- "Auto-repair improves reliability." Deterministic repair improves reliability. Guessing repair converts loud failures into quiet wrong answers, which is worse on every axis that matters.
- "If it parses, we are fine." Parsing establishes shape. A perfectly-shaped call to the wrong tool with well-typed arguments parses beautifully.
- "Retries are free, we already have the model call budgeted." Retries multiply latency and cost, and they change the output — a system with silent retries has a different reproducibility story than its authors believe.
- "Just ask the model to be more careful about the format." Sometimes it helps and it is never a guarantee, because the format is being produced by sampling. Guarantees come from constrained decoding or from validating what came back.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
Use a real parser rather than regular expressions, because a regular expression cannot find where a nested structure ends — that is a property of what regular languages can express, not a gap in your pattern. A parser also tells you the byte position, what it expected and what it found, which is what makes a repair prompt precise. Repair only what the grammar makes obvious, such as a trailing comma. Never complete a truncated value, because completing it means choosing it.
practical
Record four things on every failure: the raw output, the byte position, the expected-token set, and the failure class. The class is what decides what happens next, and the classes are not interchangeable — a parse failure is worth retrying, a policy denial never is. Cap retries per field rather than per request; two failures on the same field means your schema or description is wrong, not that the model needs another go. Chart failure position against output length, because a cluster at the end is a token-limit problem wearing a format problem's clothes.
advanced
The subtle risk in error recovery is that it succeeds too well. A recovering parser produces a tree from input it did not fully understand, and that tree is indistinguishable downstream from one produced by a clean parse — the same hazard a compiler has when it recovers from a syntax error and then reports a cascade of nonsense type errors. Compilers manage it by marking recovered subtrees as error nodes and suppressing further diagnostics that involve them, and an agent system needs the same discipline: a plan that required recovery should carry that fact, and the decision to execute it should be allowed to depend on it. The stronger version of the same idea is to refuse to execute a recovered plan at all unless a human has seen the rendered version. That is not paranoia; it is the recognition that recovery is a heuristic reconstruction of what the author probably meant, and executing a guess about intent is exactly the thing this module exists to prevent.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
If you were asked this in an interview
- Why can a regular expression not reliably extract a JSON object from a model's response?
- Which malformed outputs would you repair automatically, and which would you refuse? Justify the line.
- Your agent retries on malformed output. What could go wrong with that loop, and what would you record to notice?