Toolsconstrained decodingJSON modePydanticzodschema validation

Structured Outputs

Constrain the model to emit JSON that matches a schema, then parse it into typed objects — the right abstraction when you need data, not actions.

Interview question
Progress

What structured output is

Structured output means the model's response is a JSON document conforming to a schema you supply, instead of free prose. Providers implement this with constrained decoding: at each token the sampler masks out tokens that would make the output invalid under the grammar derived from the schema. The result is syntactically valid JSON with the right keys and types on every call — not "usually".

This is distinct from asking nicely ("respond only in JSON") and from JSON mode without a schema. The former fails a few percent of the time; the latter guarantees parseable JSON but not your shape. Schema-enforced output guarantees both.

What it does not guarantee is semantic correctness. A {"age": 250} is valid under {"type":"integer"}. Constrained decoding is a syntax guarantee; validation of meaning is still yours.

Structured output vs tool calling

Both produce schema-conforming JSON, so people conflate them. The difference is intent. A tool call says "please run this"; the loop continues after execution. A structured output says "here is the answer"; the turn is over. On the abstraction ladder from Choosing the Right Abstraction, structured output is one rung lower and correspondingly cheaper and more predictable.

Pick structured output when the model's job is extraction, classification, scoring or transformation and the output goes straight into your program. Pick tool calling when the model must decide whether and which action to take, or needs the result of an action to finish.

  • Extract {name, email, company} from a signature block → structured output.
  • Classify a ticket into one of eight categories with a confidence → structured output.
  • Decide whether to look up the order, refund it, or escalate → tool calling.
  • Produce a plan as a list of typed steps for a workflow to execute → structured output feeding a deterministic executor (Workflow State Graph).

Typed parsing with Pydantic and zod

The workflow is: define the type once, derive the JSON schema from it, send the schema to the model, parse the response back through the same type. The type is the single source of truth; the schema in the prompt cannot drift from the parser.

Both libraries also run the semantic validation that constrained decoding cannot: ranges, regexes, cross-field rules. A validation failure is a typed error you can catch and re-prompt with (Argument Validation).

Pydantic model → JSON schema → parsed, validated object.
1from pydantic import BaseModel, Field, ValidationError
2from typing import Literal
3 
4class TicketTriage(BaseModel):
5    category: Literal["billing", "bug", "feature", "other"]
6    severity: int = Field(ge=1, le=5)
7    summary: str = Field(max_length=120)
8    needs_human: bool
9 
10schema = TicketTriage.model_json_schema()
11 
12def triage(text: str) -> TicketTriage:
13    raw = chat_structured(  # provider call with response schema
14        system="Triage the support message.",
15        user=text,
16        schema=schema,
17    )
18    try:
19        return TicketTriage.model_validate_json(raw)
20    except ValidationError as e:
21        # syntactically valid JSON but semantically wrong (e.g. severity=7)
22        raise RuntimeError(f"model output failed validation: {e}") from e

The same pattern in TypeScript

zod plays the identical role. z.toJSONSchema (zod 4) or a helper library produces the provider schema; schema.parse gives you a typed value or throws a ZodError listing every violated path — ideal for feeding back to the model verbatim.

Keep the schema small. Every property costs tokens on the way in (the schema) and on the way out (the value). Ten well-chosen fields beat forty speculative ones, and models fill short flat schemas more accurately than deep ones.

1import { z } from 'zod'
2 
3const Triage = z.object({
4  category: z.enum(['billing', 'bug', 'feature', 'other']),
5  severity: z.number().int().min(1).max(5),
6  summary: z.string().max(120),
7  needsHuman: z.boolean(),
8})
9type Triage = z.infer<typeof Triage>
10 
11async function triage(text: string): Promise<Triage> {
12  const raw = await chatStructured({ user: text, schema: z.toJSONSchema(Triage) })
13  return Triage.parse(JSON.parse(raw)) // throws ZodError with per-field issues
14}

Key points

  • Constrained decoding guarantees syntactically valid, schema-shaped JSON — not semantically correct values.
  • Structured output ends the turn with data; tool calling continues the loop with an action.
  • Define the type once (Pydantic / zod), derive the schema from it, parse through it.
  • Semantic validation (ranges, regexes, cross-field rules) is still your responsibility.
  • Small flat schemas are cheaper and filled more accurately than deep nested ones.
  • Structured output is usually the right first step before reaching for tools or agents.

When to use — and when not to

Use it when
  • Extraction, classification, scoring, or transformation whose result is consumed by code.
  • Replacing brittle regex parsing of free-text model output.
  • Producing a plan or intermediate representation for a deterministic executor.
  • Building eval datasets where labels must be machine-comparable.
Avoid it when
  • The model needs to take actions or fetch data to finish — use Tool Calling Basics.
  • The answer is genuinely prose for a human; forcing JSON degrades quality.
  • Schema has dozens of optional fields; the model will hallucinate to fill them.
  • You are on a provider without schema enforcement and treat "JSON mode" as if it were.

Failure modes

  • Valid JSON, wrong values: severity: 5 on every ticket because the description never defined the scale.
  • Schema drift: prompt schema and parser type maintained separately and disagreeing.
  • Required field the model cannot know; it invents a plausible value instead of leaving it null.
  • Deep nesting or unions unsupported by the provider silently downgraded to plain JSON mode.
  • Treating a parsed object as trusted and passing it to SQL or a filesystem without checks.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy

The cheapest rung above a plain LLM call; single round trip, fully inspectable output.

← Tool SchemasArgument Validation →