.json: JSON Schema, enforced in your inference engine · .txt

Does your provider break your tool calls?
Most structured-generation engines handle types and required fields, then quietly skip the rest of your schema. The output still parses, so you trust it, and the wrong values flow into the rest of your system. .json enforces the whole schema while the model writes, if/then/else included, or refuses it up front and tells you why. It runs inside your own inference engine, with no slowdown.
A Rust library with Python bindings, for teams serving their own models.
Types and required fields are easy, and every engine handles them. The rules are harder: a German invoice needs a German VAT number, a refund needs a reason. In JSON Schema you write those with if/then/else, and neither OpenAI's nor Anthropic's structured outputs will take it.
Some open-source engines accept a keyword and then don't enforce it. XGrammar, a common default in inference servers, ignores a pattern on a date and lets any date through. Either way the output parses, and you find out it's wrong later. .json enforces the rules as the model writes.
A mask can go wrong in two ways. It can let an invalid token through, and you get broken output. Or it can block a valid one, and the model quietly loses options it should have had.
We compared .json, XGrammar and llguidance token by token on 5,047 schemas from JSONSchemaBench, at 431,966 positions. We found no errors of either kind in .json. XGrammar let 3.4 million invalid tokens through; llguidance blocked 1.5 million valid ones.
.txt's benchmark with .json v0.0.49, XGrammar 0.1.32 and llguidance 1.6.1, on the Qwen2.5-7B-Instruct vocabulary. Counts are lower bounds on wrong tokens, not failed outputs. The departures from the specification each library documents are left out.
OpenRouter publishes how often each provider's structured outputs fail. Across 40 open-weight models in the week of 3 to 9 November 2026, half of all provider and model pairs failed more than 1.6% of requests. On a typical model, the worst provider failed 12 times as often as the best, and for 9 of the 40 models no provider got under 1%.
The weights are the same everywhere. What changes is how they're served.
OpenRouter's published structured-output error rates, 3 to 9 November 2026: of the requests asking for a structured output, the share OpenRouter counts as failed. The chart shows the 25 models with the most providers and the 37 providers serving at least three of them: 296 provider-model pairs with 1,000 or more such requests a day.
.json covers most of JSON Schema 2020-12, including what OpenAI's and Anthropic's structured outputs leave out: tuples, oneOf, not, if/then/else, dependentRequired. When it can't enforce a keyword, it refuses the schema rather than ignoring the keyword. The table lists only the rows where the three differ. See every supported keyword.
✓ supported, ◐ partial, ◇ available on request, ✗ unsupported. Only features where at least one provider differs are listed: basic types, items, pattern and $defs / $ref are supported by all three. Last verified 26 February 2026.
.json is a drop-in replacement for the structured-output backend your engine ships with, and it takes about a dozen lines. Before generation starts, it compiles your schema against the model's vocabulary; you do that once and keep the result on disk. Then each step is two calls: mask the logits, so only the tokens the schema allows can be sampled, and tell .json which token you picked.
The model can't write an invalid character, so there's nothing to check afterwards and nothing to retry. Your model, your sampler and your stopping rules stay where they are.
Your JSON Schema and the model's tokenizer.
The schema decides which tokens are allowed. The model chooses among them.
JSON that matches the schema, returned by your engine as usual.
vLLM · SGLang · TensorRT-LLM · ONNX Runtime GenAI.
A Python wheel, and a Rust library for runtimes that aren't written in Python. Each comes with patches for your inference engine that make .json its structured-output backend.
No. Outlines is our open-source library, and it stays open source. .json is a commercial product built on a separate engine. It covers more of JSON Schema, and we maintain and support it in production.
.lambda is for tool calls. It knows each model's own tool-call format, enforces it while the model writes and hands your engine a parsed call. .json is for output that has to match a JSON Schema you give it. If your model calls tools, start with .lambda.
Compilation fails and names the keyword. Reject the request or handle it yourself, but don't fall back to unconstrained generation: that's how bad output gets through. The docs list every keyword .json supports.
No. Working out which tokens are allowed is separate from applying the mask, so your engine can run it alongside the model's forward pass.
Yes, and that’s where .json’s speed shows most. Checking a run of draft tokens needs a mask for each of them in the same step, so a slow mask adds up fast.
No. .json guarantees the output matches the schema. It can still hold the wrong fact or the wrong choice: the schema decides what can be written, and the model still decides what is.
Yes. Compile a schema against a tokenizer once and save it to disk. Rebuild it when you move to a version of .json whose format has changed.
Send the schemas you serve, the model's tokenizer and your engine version. An engineer will run them through .json and tell you what compiles, what doesn't, and what your current setup has been skipping.