Structured generation produces model output that follows a defined machine-readable format or schema. It can use ordinary prompting, constrained decoding, or a provider's structured-output interface. The guarantee depends on the mechanism and its documented limits.
Keep three properties separate: syntactic validity means the output can be parsed; schema validity means it satisfies the supported field/type constraints; semantic correctness means the values are right for the task. None of these alone authorizes an external action.
Compare the available mechanisms
| Mechanism | What it aims to enforce | Remaining responsibility |
|---|---|---|
| “Return JSON” in a prompt | A requested output style | Parse, validate and handle deviations |
| JSON mode | JSON syntax under the provider's contract | Required fields and business semantics |
| Schema-constrained output | Supported schema constraints during generation | Completion status, unsupported constraints and factual checks |
| Tool/function calling | A tool name and argument structure | Authorization, execution and result handling |
| Application validation | Explicit checks after generation | Recovery when a check fails |
Do not copy one provider's parameters into another API. Even features with similar names can support different schema subsets, nesting, streaming and refusal behavior. For example, Claude's current documentation describes schema constraints alongside explicit refusal and token-limit exceptions. These still require application handling. Claude structured-output documentation.
How constrained decoding works
A constraint engine tracks which continuations remain legal under the chosen grammar or schema. It masks incompatible next-token choices before sampling. Tokens may encode multiple characters, so this is not simply a choice between individual punctuation marks.
Regular expressions can describe many flat patterns. Context-free grammars can represent recursive structures such as nested expressions. A schema compiler translates supported structural constraints into a form the decoder can enforce; it may reject or leave some constraints for post-validation. Guided-generation research describes efficient ways to apply these constraints. Willard and Louf.
A small positive or negative logit bias merely changes probabilities. It is not equivalent to eliminating every invalid continuation. Grammar enforcement also cannot establish that a cited invoice exists or that a requested recipient is permitted.
Work through an extraction contract
Suppose an invoice contains a USD total of 125.40 on page 2. An illustrative extraction result is:
{
"status": "found",
"invoice_id": "INV-208",
"amount_minor": 12540,
"currency": "USD",
"evidence_ids": ["page-2-total"]
}
The contract defines amount_minor as integer currency minor units and defines which currencies are accepted. Do not assume every currency has two decimal places. If the total is absent or conflicting, return an explicit missing or conflicting status with nullable values instead of inventing a number to satisfy a required field.
The application checks:
- The transport and generation completed normally.
- The payload parses and follows the supported schema.
- The currency, amount and status combination follows business rules.
- Evidence references resolve to the permitted source and support the value.
- Any downstream action has separate identity, permission and approval checks.
A valid extraction of an invoice amount is not permission to pay it.
From model output to a safe action
Read diagram source
flowchart LR
M[Model output] --> F[Check completion and refusal]
F --> S[Parse and validate schema]
S --> V[Verify facts and business rules]
V --> A[Authorize exact action]
A --> E[Execute with operation identity]
E --> R[Record actual result]
Tool calling is a proposal to invoke a tool. The application chooses whether and how to execute it. Parallel calls are appropriate only when their dependencies and permissions permit it. Reading two independent records may be parallelizable; spending a balance and then reporting the new balance requires ordering and consistency rules.
For streamed output, partial JSON or partial arguments may be incomplete. Do not execute a consequential call from a prefix before the whole proposal is validated. A reconnect or retry must not duplicate a previously executed operation.
One pass or multiple stages?
| Design | Useful when | Tradeoff |
|---|---|---|
| One constrained extraction | Evidence fits and fields are related | One error can affect several related fields |
| Independent field groups | Groups use separate evidence or expertise | Merge logic and cross-field consistency |
| Extract evidence, then normalize | Raw evidence needs careful preservation | Extra call, latency and possible handoff loss |
| Deterministic parsing after extraction | Formatting or arithmetic is precisely defined | Requires an explicit supported input contract |
There is no universal rule that fifty fields require a free-form first pass. That pass can omit evidence or introduce an unsupported fact before conversion to JSON. Evaluate complete records, field accuracy, missing-value behavior and source alignment for each design.
Repair failures without hiding them
Return a short validation error that identifies the failed contract, with sensitive details removed. For example: “amount_minor must be an integer or null; evidence is required when status is found.” Do not send a full production traceback containing secrets or unrelated records.
Use a bounded retry policy. A formatting correction may be recoverable; contradictory source data needs clarification or review. Track first-pass validity, eventual success, retries, total cost and severe semantic errors. Counting only the final valid JSON hides expensive and unreliable behavior.
Interview practice
Q1: Does structured output eliminate hallucinations?
No. It can constrain the representation while the values remain unsupported. I validate important facts against source evidence and allow missing or uncertain outcomes. A beautifully formed object can describe a nonexistent invoice.
Q2: Why validate again if the provider supports strict schemas?
I must handle incomplete responses, refusals, documented schema limitations and business constraints beyond the decoder. Validation also protects the application when models, SDKs or schemas change. The checks should reflect the actual contract, not assume every returned text block is successful data.
Q3: How do you avoid forcing invented values?
Define statuses and nullable fields for absent or conflicting evidence. Explain the relationship between status, values and evidence IDs. Evaluate abstention quality alongside extraction accuracy, since returning null for every difficult case is not useful either.
Q4: Can five tool calls always run in parallel?
No. Check data dependencies, write conflicts, rate limits and authorization. Parallel generation of arguments does not prove that executing the actions concurrently is correct. A tool result may be needed to construct or approve the next action.
Q5: What is dangerous about retrying a failed tool response?
The tool may have completed the side effect before the response was lost. Use stable operation identity, idempotent APIs where available and reconciliation of uncertain outcomes. Retrying generation and retrying execution are separate decisions.
Q6: Would you split a large schema into smaller extractions?
I would first measure field-level and record-level errors. Splitting can improve focus, but it adds calls and may break relationships across fields. Preserve source references and apply a final consistency check before comparing quality and cost with the single-pass baseline.
Final notes
Recall card: Complete → parse → validate structure → verify meaning → authorize → execute. Structured generation reduces interface errors; the surrounding system establishes correctness and authority.
Related: prompt fundamentals, context engineering, prompt injection.