Asking a model politely for JSON and parsing whatever comes back was standard practice for a while, and it failed regularly - markdown fences, explanatory preambles, trailing commas, truncated objects. Constrained decoding and tool calling solved the parsing problem properly. What remains is a design problem: schemas that are easy for a model to fill correctly.
01Use the mechanism, not the prompt
Requesting JSON in the prompt is a suggestion the model usually follows. Schema-constrained output is a guarantee enforced during generation - the decoder can only emit tokens that keep the output valid against the schema, so malformed JSON becomes structurally impossible rather than merely unlikely.
This eliminates an entire category of production errors, including the ones that appear only under load or with unusual inputs. If your provider supports constrained decoding or tool calling with a schema, use it. Prompt-based JSON requests should be considered legacy.
02Schema design shapes accuracy
A valid schema guarantees the shape, not the content. The model will produce something that fits, and if the schema is ambiguous it will fit incorrectly. Field naming, descriptions and constraints materially affect whether the values are right.
Use descriptive names rather than abbreviations - the model reads them as instructions. Add descriptions to fields whose meaning is not obvious from the name. Prefer enums over free strings wherever the valid set is known, which removes an entire class of near-miss values. And flatten deep nesting where you reasonably can, because deeply nested structures produce more errors.
- Name fields descriptively - the schema is part of the prompt.
- Add descriptions for anything ambiguous or domain-specific.
- Use enums instead of free-text wherever the value set is known.
- Keep nesting shallow; flatten where the structure allows.
- Mark fields optional only when absence is genuinely meaningful.
03Give the model somewhere to put uncertainty
A schema requiring a value forces the model to produce one even when the input does not support it. Extracting an invoice total from a document that has no total will produce a plausible number, because the schema demanded a number and there was no alternative.
Provide an explicit escape: a nullable field, a confidence indicator, or a separate flag for insufficient information. Models will use these when available, and a null with a stated reason is far more useful than a confident fabrication. This single design choice removes a large share of hallucinated field values.
04Tool calling is structured output with consequences
Tool or function calling uses the same constrained generation, but the output triggers an action rather than being displayed. That raises the stakes considerably - a malformed extraction is a bad record, while a malformed tool call might delete something.
Design tool interfaces the way you would design an API for an unfamiliar developer who will not read the documentation carefully. Narrow parameters, strong validation, no ambiguity between similar tools, and clear names. Two tools with overlapping purposes will be confused; merge them or make the distinction explicit in the descriptions.
| Principle | Why | Anti-pattern |
|---|---|---|
| One clear purpose per tool | Ambiguity causes wrong selection | A generic 'query' tool with a mode flag |
| Validate server-side | Schema conformance is not authorisation | Trusting parameters because they parsed |
| Return structured errors | Model can correct and retry | Raw stack traces or empty failures |
| Keep the tool count modest | Selection accuracy degrades with size | Forty tools in one context |
| Confirm destructive actions | Mistakes should be recoverable | Direct delete with no confirmation |
05Validate anyway
Constrained decoding guarantees the output matches the schema. It does not guarantee the values are correct, permitted, or sane. A schema-valid tool call can still reference a record the user cannot access, request a negative quantity, or specify a date in the past.
Treat every tool call as an untrusted request from a client, because that is what it is. Authorise against the requesting user's permissions, validate business rules, and check referential integrity. The schema is the first gate, not the only one, and skipping the rest is how prompt injection turns into data loss.
06Handle the remaining failure modes
Two problems survive constrained decoding. Truncation: hitting the output token limit mid-structure produces incomplete output, which matters most for long lists. Set generous limits, request pagination for large results, and detect truncation explicitly rather than parsing a partial object.
The second is repetition, where a model generating a list continues past what the input supports, inventing entries to fill an implied quota. Setting maximum array lengths in the schema and explicitly permitting empty arrays both help. Log the distribution of result counts - a suspicious cluster at your maximum suggests the model is padding.
Topics
Aarav Patel
Principal Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.