There is a genre of production incident that should be extinct by now and is not. The model was asked for JSON. The model returned JSON, wrapped in a code fence, with a friendly sentence first. Or real JSON with one field renamed. Or valid JSON that stops mid-object at the token limit. The parser throws, the retry fires, the retry does it again, and someone is reading stack traces at 2 a.m. because a text generator was asked nicely to behave like an API and, one time in two hundred, declined.

Every one of those incidents is optional. Constrained decoding has been broadly available for a couple of years, it costs roughly nothing, and it moves the guarantee from "the model usually complies" to "non-conforming output cannot be sampled." The reason I keep meeting pipelines that still parse-and-pray is not that the feature is obscure. It is that teams think of output format as a prompting problem, and this stopped being a prompting problem the day the samplers learned grammar.

The Ladder: Pray, JSON Mode, Schema

There are three rungs, and teams routinely believe they are one rung higher than they are.

Rung one: prompt-and-pray. You describe the format in the prompt, maybe with examples, and parse what arrives. Compliance is high and probabilistic, which in production arithmetic means failure is scheduled: at 99.5 percent compliance, a pipeline doing 100,000 calls a day takes 500 format failures a day, each one a retry, a fallback, or a corrupted record.

Rung two: JSON mode. The provider guarantees syntactically valid JSON. This kills the code fences and the chatty preamble, and guarantees nothing about shape: fields can be absent, renamed, nested wrong, or invented. Valid JSON with the wrong shape is arguably worse than a parse error, because it fails past the first try-catch and dies deeper in the system, where the error message no longer points at the cause.

Rung three: schema-constrained decoding. You hand over a JSON Schema; the runtime compiles it into a grammar; at every decoding step, tokens that would violate the grammar are masked out before sampling. The model literally cannot emit a non-conforming output, the way a chess engine cannot play an illegal move. Required fields arrive, enums come from the allowed set, types are correct, by construction. Parsing stops being error handling and becomes deserialization.

The mechanism matters because it explains both the strength and the cost. The mask does not make the model smarter; it narrows what the model can say next. Guarantees about shape become absolute. Everything about the values stays exactly as probabilistic as it was.

What the Guarantee Does Not Cover

Schema conformance is where a specific failure mode moves in: the confidently formatted wrong answer. An unconstrained model that does not know the answer often rambles, hedges, or drifts off format, and the ugliness is a signal your pipeline can catch. A constrained model has no way to look confused. It will fill every required field with something type-correct, pick an enum value because the grammar demanded one, and hand you a beautifully shaped object that is wrong inside.

Two habits keep this honest. First, every closed set gets an escape hatch: an unknown or none_of_these option, an explicit null for extractions that found nothing. A schema without an out is an instruction to invent, and the model will comply. Second, shape guarantees do not retire semantic checks: the assertion layer of your evals still checks that the date is a plausible date and the cited span exists in the source. Structured outputs eliminate the parse failures so your assertions can spend their budget on meaning.

The Schema Is a Prompt: Design It Like One

The runtime reads your schema as a grammar. The model also reads it as context, which makes schema design a quiet branch of prompt engineering, and most schemas I review were written for a database, not for a model.

Field names and descriptions are instructions. The model sees them, and they steer it. A field named sentiment with a description that defines the boundary between neutral and negative outperforms a comment in your codebase that the model never sees. Write descriptions the way you would write rubric anchors.

Prefer enums to free strings wherever the set is closed. Every free-text field is a place where downstream code will meet creative spelling. If the value feeds a switch statement, it should be an enum in the schema, not a convention in the prompt.

Keep it flat and small. Deep nesting and sprawling optionality cost attention and multiply the ways a semantically confused output can still validate. If the task needs forty fields, it is usually several tasks.

Order fields for reasoning. Generation is sequential, so the schema's field order is the model's thinking order. Put a reasoning field before the verdict fields and the model works through the evidence before committing; put the verdict first and the reasoning field becomes a post-hoc rationalization of a guess. This single ordering choice is the difference between constrained output degrading accuracy and improving it, and it is why judge verdicts should always be schema-constrained with the rationale up front.

Where It Composes

Once you see structured output as a primitive, it shows up everywhere in the stack. Tool calls are structured outputs with a dispatcher attached: the inputSchema on every MCP tool definition is exactly this contract, and malformed tool arguments are the same disease as malformed JSON, cured the same way. Extraction pipelines, classifiers, and routers are the obvious consumers. Judges emit calibrated verdicts as schemas. And in agent systems there is a security dividend most teams never notice: an output constrained to an enum and a bounded summary is a far narrower channel for a hijacked model to smuggle anything through than free text with room for a URL. The schema is a leash, and sometimes the leash is the point.

Where does it not belong? Prose meant for humans. An essay, an email draft, a support reply wants rhythm, not a grammar mask, and wrapping conversational text in a one-field schema buys you nothing but ceremony. The rule of thumb I use: if a machine consumes it, constrain it; if a person reads it, let it breathe.

The reliability conversation in this field keeps reaching for the expensive levers: bigger models, more retries, judge pipelines, agent supervisors. Meanwhile the cheapest lever sits in the request options, deployed in an afternoon, removing an entire failure class by construction. Take the free win first. Then spend your cleverness on the failures that remain, which are the interesting ones anyway.

FAQ

What are structured outputs?
A decoding mode where output is constrained to a schema by construction: tokens that would violate the structure are masked before sampling, so the output cannot fail to parse. Stronger than JSON mode, which guarantees syntax but not shape.

JSON mode vs. structured outputs?
JSON mode guarantees valid JSON; fields can still be missing, renamed, or invented. Schema-constrained output guarantees your shape: required fields, correct types, enums from the allowed set. Validation moves from your error handler into the sampler.

Do structured outputs guarantee correct answers?
No. Shape, not truth. Constrained models produce confidently formatted wrong answers where unconstrained ones ramble visibly, so schemas need escape hatches and the values still need assertions and evals.

When should you skip them?
Prose for humans. And order a reasoning field before verdict fields in any schema that involves judgment, so the constraint helps accuracy instead of taxing it. Everything machine-consumed should be constrained by default.