# Constrained Decoding: Guaranteeing Valid JSON Output from LLMs

> Explore how constrained decoding ensures structurally valid JSON from LLMs, contrasting it with prompting and examining schema compliance vs. extraction correctness.

Published: 2026-11-19

Canonical: https://sygnet.ai/blog/constrained-decoding-guaranteeing-valid-json-output-from-llms

---

## Key takeaways

- Constrained decoding, not prompting, is what guarantees structural validity: the schema is compiled into a grammar and every token that would break it is masked to probability zero ([OpenAI](https://openai.com/index/introducing-structured-outputs-in-the-api/)).
- Schema compliance and extraction correctness are different metrics: GPT-5.4 hits a 99.97% JSON pass rate yet only 0.486 perfect responses, roughly a 51-point gap ([Structured Output Benchmark](https://arxiv.org/html/2604.25359v1)).
- Strict modes only cover a subset of JSON Schema, and coverage collapses on real-world schemas: OpenAI's strict mode drops to 29% empirical coverage on the GitHub Easy dataset of JSONSchemaBench.
- Use JSON for business fields and ALTO/hOCR XML for coordinate-level page geometry; they answer different questions and you often need both.

## Why does "return JSON" in the prompt keep failing?

Because prompting produces a learned tendency, not a guarantee. A strong model asked to return JSON will occasionally emit malformed output regardless of its capability, because free-form generation has no hard guarantee of matching a schema; constrained decoding makes invalid output structurally impossible rather than merely unlikely. The failure modes are familiar to anyone who has shipped an extraction pipeline: markdown fences, a stray brace, a trailing comma, a helpful preamble.

OpenAI's own framing drew the line clearly at launch: JSON mode improves reliability for generating valid JSON, but it does not guarantee that the response will conform to a particular schema. The cost of the gap is not just parse errors. Retries stack latency and spend, and a repair pass can still let a subtly wrong payload through. The deeper argument is compounding: a five-step pipeline where each step is 93% reliable produces correct end-to-end results about 70% of the time. For document AI, where classification, field extraction and validation chain together, that arithmetic decides whether the system is usable.

> The prototype retries; the production system constrains generation.

## How does constrained decoding actually enforce a schema?

It intervenes between the forward pass and sampling. A language model produces a probability distribution over its whole vocabulary at every step; constrained decoding computes which tokens are legal given the grammar and the text so far, sets every other token's probability to zero, then samples. If a field is an enum of `approved` and `rejected`, the model is structurally incapable of producing "maybe" in that slot.

Crucially, the relative ranking among legal tokens is preserved: if the model assigns 0.3 to token A and 0.1 to token B and both are valid, the constrained distribution keeps the 3:1 ratio, and intervention only happens when the model would choose something structurally invalid.

The engineering difficulty is speed. Tokenizers have over 100,000 entries, a nested schema needs a pushdown automaton rather than a regex, and the mask must be ready before the next forward pass finishes. The main engines solve this differently: XGrammar from the MLC team claims up to 100x speedup over prior solutions and is the default structured generation backend in vLLM, SGLang, TensorRT-LLM and MLC-LLM, while llguidance, the Rust core behind Microsoft's Guidance, reports around 50 microseconds of CPU time per token mask for a 128k-token tokenizer.

## Which enforcement mode should a data team pick?

Pick by where you run inference, not by model leaderboard position. If you call a hosted API, provider strict mode is the only hard guarantee available. If you control the runtime, open engines give you wider schema coverage and better control over caching.

| Approach | Guarantee | Known limits |
|---|---|---|
| JSON mode (`json_object`) | Syntactically valid JSON only | No schema adherence; prompt must mention JSON or the API errors |
| OpenAI strict (`json_schema`) | 100% schema adherence in OpenAI's own evals | No `allOf`, no `if/then/else`; every key required with `additionalProperties: false` |
| Anthropic tool use | Trained adherence, not decode-time masking | No hard API-level guarantee the way `strict: true` gives |
| Gemini `responseSchema` | 100% compliance on supported schemas, 86% empirical coverage | Nesting beyond 3-4 levels pushes failure rates to 1-2% |
| Self-hosted (Outlines, XGrammar, llguidance) | Grammar-enforced, full CFG power | Compile cost: Outlines compiles grammars in 3-8 seconds vs Guidance's near-zero |

Budget for first-request compilation. OpenAI now says only fine-tuned models pay extra latency on the first request with a new schema, while Anthropic compiles the schema into a grammar on first use with a documented 24-hour cache, a 180-second compile timeout, and a 400 error for over-complex schemas. If you are weighing hosted strict modes against a self-hosted stack, our [build vs buy IDP breakdown](https://sygnet.ai/build-vs-buy-idp) covers the operational side of that decision.

## Does 100% schema compliance mean the extraction is correct?

No, and conflating the two is the most expensive mistake in document AI. Structured outputs guarantee shape, not substance: a response can have every required field, correctly typed, with an enum value that is technically valid and semantically wrong for the input.

Benchmark numbers make the gap measurable. On the Structured Output Benchmark, structural metrics (JSON Pass, Path Recall, Type Safety) cluster above .95 for most models, yet Value Accuracy spans a 13.7-point range from 0.693 to 0.830. The authors are blunt that the challenge is accurate value extraction from noisy OCR and complex multi-speaker transcripts, not structural compliance.

Schema-constrained OCR benchmarks show the same decoupling at model level: best at OCR does not mean best at schema-following, the best model by flat-text CER is not the best JSON extractor, and the winner on schema-constrained extraction was a model that barely cracked the top 14 on plain OCR. Measure per field, against labelled documents. Our [extraction accuracy evaluation checklist](https://sygnet.ai/checklists/extraction-accuracy-evaluation) lays out the field-level protocol; for tables specifically, structural edit-distance metrics like GriTS and TEDS are covered in our [table extraction metrics guide](https://sygnet.ai/blog/table-extraction-metrics-grits-teds-semantic-evaluation).

> Schema validity is a settled syntactic problem and an unsettled semantic one.

## How should you design a schema for document extraction?

Keep it flat, keep it narrow, and encode known answer sets as enums. OpenAI's documented limits are 10 levels of nesting and 5,000 object properties per schema, and for tree-shaped data flattening to a list with parent IDs beats nested children: more reliable, easier to validate, easier to render. That advice maps directly onto invoice line items, claim lines, and lease clause lists.

Three concrete rules:

**Drop unused fields.** Every field in the schema is generated, so remove anything you will not consume. There is a payoff beyond latency: a constrained schema emits only the requested fields, with no preamble or markdown fence, so output is frequently shorter than the free-text equivalent, and since generated tokens are the expensive half of an inference bill, a tighter schema can lower cost.

**Handle optional fields deliberately.** Under OpenAI strict, optional fields are a union with null. A workable alternative is to include all fields and have the model populate unneeded ones with placeholders, deleting them in post-processing.

**Watch the quality tax.** Constraints can push a model off its preferred path. Pulse AI reports experiments tracking perplexity during generation to flag extractions where the model was "fighting" the constraints, routing those high-perplexity results to human review. They also found that generating with minimal constraints then restructuring in a second pass often outperforms direct constrained generation on reasoning-heavy tasks.

## When should OCR output be XML instead of JSON?

When the consumer needs page geometry and provenance rather than business fields. ALTO, maintained by the Library of Congress, is a layout format holding word- and line-level bounding-box coordinates alongside recognised text; it answers where on the image this text is and what it reads, not what the document is. It is an open XML schema originally developed by the EU-funded METAe project, describing placement, size and style of text plus margins, headings, columns and illustrations, and maintenance transferred to the Library of Congress in August 2009.

ALTO carries per-unit confidence natively: the schema defines an estimated percentage of OCR accuracy from 0 to 100 and a page confidence level between 0 (unsure) and 1 (sure), plus word-level confidence on each string. Since version 4, ALTO added `<Shape>` polygons and polyline baselines, closing the functional gap with PAGE XML for handwritten material.

hOCR is the lighter option: it embeds layout and transcription metadata directly into semantic HTML/XHTML using microformats and class attributes rather than defining a new XML schema. Tesseract generates native ALTO XML and hOCR output, and third-party wrappers convert to PAGE XML. For auditable pipelines, keeping coordinates alongside extracted values is what lets a reviewer verify a field against the original pixels, a requirement we treat as baseline in our [RAG document ingestion checklist](https://sygnet.ai/checklists/rag-document-ingestion).

## What belongs in the validation layer after constrained decoding?

Keep a structural validator as a tripwire, then add the checks a schema cannot express. With a correct engine the structural re-check should never fail, so a failure there is a high-signal alert that the grammar or the engine is wrong, not the model.

Above that sits semantics: that an extracted total equals the sum of line items, that a date falls in a plausible range. These are the rules that catch the valid-but-wrong class of error. Refusals need their own path too: if the model judges a request unsafe it can decline the schema entirely and the response carries a refusal flag, so any production pipeline needs a code path that checks for this explicitly.

Finally, instrument retries. Modes that hit 100% with retries hidden behind the SDK are paying on the retry line item; the right unit is schema validity per call after N retries, at cost C. Sygnet runs document extraction with schema-constrained output and field-level confidence, and publishes its [security and compliance posture](https://sygnet.ai/security) for teams that need to document where documents are processed.

## FAQ

### Can I get guaranteed JSON from an open-weight VLM running on-prem?

Yes, provided your serving stack exposes the sampler. Grammar constraints require sampler access, so support is a property of the serving stack, not the model weights. XGrammar ships as the default backend in vLLM, SGLang and TensorRT-LLM. On the schema-constrained OCR benchmark, an 8B model running on a single A10G with 24 GB VRAM scored 0.730, beating olmOCR-7B at 0.714 and Claude Sonnet 4.5 at 0.709, evaluated with xgrammar structured generation.

### Does enforcing a schema hurt extraction accuracy?

Less than commonly feared, but it varies by model. On the Structured Output Benchmark, schema constraints had a smaller effect on grounded extraction than on structural reliability, with the direction varying by model: Gemini-2.5-Flash and Gemini-3-Flash-Preview saw JSON Pass rise while GPT-5.4 saw it fall slightly. Risk concentrates in deeply nested or constraint-dense schemas, which is the practical argument for flattening before you blame the model.

### Why does my schema get rejected by one provider but accepted by another?

Because each strict mode supports a different JSON Schema subset. Nesting is capped, root objects cannot use `anyOf`, and Azure's implementation adds its own restrictions beyond OpenAI's base list. Assume no parity: a schema that works on OpenAI strict will not work identically on Anthropic tool use or Gemini responseSchema. Test each target separately, and prefer rejection at integration time over silent downgrade to unguided generation.