Stop Treating LLM JSON as a Polite Suggestion: Schemas, Validation, and Repair Loops

If you are building AI-powered automation, you have likely encountered the “mostly valid” JSON trap. You prompt the model, it returns a JSON blob, and your code tries to parse it. Sometimes it works. Sometimes it fails because of a trailing comma, a hallucinated field, or a truncated string. In a prototype, this is a nuisance. In production, where AI feeds databases, message queues, or downstream agents, “mostly valid” is a liability.

The industry is shifting from treating structured outputs as a convenience feature to treating them as a reliability layer. This shift is not about making LLMs smarter; it is about enforcing contracts. When we build pipelines, we need to stop asking models to be polite and start forcing them to be compliant.

The Myth of ‘Mostly Valid’ JSON

The root of the problem is a fundamental misunderstanding of what an LLM does. An LLM is a probability engine, not a database. When you ask for JSON via a prompt, you are making a suggestion. The model will do its best to comply, but it is not bound by the structure you defined in your head.

This leads to real-world pain points that break downstream systems. We see trailing commas in the last object of an array. We see fields that the schema didn’t ask for but the model decided were important. We see truncation when the context window fills up mid-generation. These aren’t edge cases; they are the default behavior of autoregressive generation.

The difference between prompt-based JSON and API-enforced structured outputs is the difference between a handshake and a contract. Prompt-based JSON relies on the model’s instruction-following capabilities, which are probabilistic. API-enforced structured outputs rely on deterministic constraints. As noted in recent developer guides, the shift from JSON Mode to Strict Mode represents a move toward reliability. Strict Mode uses a Context-Free Grammar (CFG) engine to mask invalid tokens before generation, achieving near-perfect schema compliance for supported models like GPT-5.2 [1].

However, even with strict enforcement, we must acknowledge that the model can still return perfectly typed but factually wrong data. The schema ensures the shape is correct; it does not ensure the content is true. This distinction is critical for operators who need to build resilient pipelines.

Schema-First Development: The New Standard

To build reliable automation, we must adopt schema-first development. This means defining the structure of the data before writing a single line of prompt text. In TypeScript, this looks like Zod schemas. In Python, it looks like Pydantic models. The goal is to have a single source of truth for what the output should look like.

Defining schemas early forces clarity. You cannot validate what you haven’t defined. By using enums and stable IDs for routing fields, we keep the output narrow and predictable. This reduces the cognitive load on the model and minimizes the surface area for errors.

OpenAI’s documentation highlights this shift explicitly. The era of relying on JSON Mode is ending; Strict Mode is becoming the default for reliability [2]. This isn’t just a technical update; it’s a philosophical one. We are moving from hoping the model gets it right to ensuring the model can only get it right.

But schema-first development has tradeoffs. It requires upfront effort. It makes prompt iteration slower because you must update the schema, the validation logic, and the prompt simultaneously. However, this friction is necessary. It prevents the drift that occurs when prompts evolve independently of the data structure they produce.

The Validation and Repair Loop

Even with strict schema enforcement, production pipelines require a validation and repair loop. This is the core automation pattern for handling LLM outputs reliably. The loop consists of four steps: Generate, Validate, Repair, and Retry.

1. Generate: The model produces a candidate JSON output.
2. Validate: The output is checked against the schema. This includes checking for syntax errors (like Markdown fences), type mismatches, and missing required fields.
3. Repair: If validation fails, the error context is sent back to the model. The model is asked to fix the specific errors identified in the previous step.
4. Retry: The repaired output is validated again. If it passes, the pipeline proceeds. If it fails, the loop repeats.

This loop is not optional for production systems. As detailed in practical guides for real pipelines, the validate-repair-retry loop is essential for handling the inherent noise in LLM outputs [3]. The key is to send specific validation errors back to the model, not just a generic “try again.” This gives the model the context it needs to correct its mistake.

We must also handle “refusals” as first-class errors. Sometimes the model refuses to generate the output due to safety filters or other constraints. This is not a parsing failure; it is a refusal. Our error handling must distinguish between these cases to route them appropriately.

Crucially, we must cap retries to prevent infinite loops. If a prompt consistently requires more than two retries, the issue is likely in the prompt or schema design, not the model’s capability. In such cases, we should log the retry rate as a key metric and investigate the root cause. Infinite token burns are a cost center, not a feature.

Observability and Versioning

You cannot improve what you cannot measure. In automation pipelines, observability is as important as the logic itself. We need to log raw output, validation errors, and retry counts as replayable events. This allows us to debug failures and track the health of our pipelines over time.

Key metrics to track include:
* Parse Rate: The percentage of outputs that are valid JSON.
* Schema Pass Rate: The percentage of outputs that pass schema validation.
* Business Pass Rate: The percentage of outputs that pass domain logic checks.
* Retry Cost: The average number of tokens spent on retries.

Versioning is equally critical. Schema changes are breaking changes. Production pipelines must version schemas and prompts together. We should record schema_version and prompt_version in every event for auditability. This enables rollbacks and sane audits when something goes wrong. If a pipeline breaks, we need to know exactly which version of the schema and prompt was used.

Practical Implementation for Builders

For operators looking to implement structured outputs, start small. Pick one workflow that causes the most downstream pain, such as ticket routing or entity extraction. Add validation and one repair retry to that single pipeline. Measure the improvement. Then expand.

Run adversarial schema tests in CI to catch drift before production. These tests should include edge cases, malformed inputs, and boundary conditions. If the pipeline fails these tests, it should not be deployed.

We must also make concrete decisions about what to automate with schemas and what to keep deterministic. Schemas are best for structured data extraction, routing, and configuration. They are less effective for creative tasks or open-ended generation. For those tasks, we should keep humans in the loop or use deterministic logic where possible.

The goal is not to eliminate LLMs from our pipelines but to make them reliable components. By enforcing schemas, validating outputs, and implementing repair loops, we can build automation that works consistently. This is not about perfection; it is about predictability.

Sources and further reading

• LLM Structured Outputs: Schema Validation for Real Pipelines
• Build iterative repair loops with Codex
• OpenAI Structured Outputs: Complete Developer Guide
• LLM Structured Outputs in Production: How to Stop JSON From Breaking Your AI Workflow
• Structured Outputs | Enforce JSON Schema in OpenRouter API
• What is Structured Data Output?