Structured output and what happens when the schema drifts
The pitch versus the reality
Every major LLM provider now sells some version of “structured output.” You give the model a JSON schema, it gives you back JSON that matches. On paper this solves the single most annoying part of building on top of these models: parsing free text and hoping the model didn’t decide to add a friendly opening sentence before the JSON blob.
In practice, structured output comes in two flavors, and the difference matters more than the marketing copy suggests.
The first is constrained decoding. The model’s output is masked at the token level so it can only produce tokens that keep the output valid against your schema. OpenAI’s strict: true mode on structured outputs works this way, as does Google’s response_schema on Gemini models. If the schema says a field is an enum with three values, the sampler literally cannot produce a fourth. This is a hard constraint at the decoding layer, not the model “trying hard” to follow instructions.
The second is prompted JSON, sometimes dressed up as “JSON mode.” The model is told, via system prompt or fine tuning, to produce JSON, and the API might validate the output is at least syntactically valid JSON. But nothing stops it from inventing a field, dropping one you required, or returning a string where you specified an integer. Anthropic’s tool use with a defined input schema sits closer to constrained behavior when you’re using Claude to call a tool, but the guarantees depend on which mode and which model version you’re pointed at, so check the current docs for whatever you’re shipping against rather than assuming.
If you don’t know which flavor you’re using, you don’t actually know what your error rate looks like. That’s the first thing to nail down before you build anything on top of it.
What “the schema drifts” actually means
Schema drift is usually one of three things happening quietly. None of them is the model deciding to rebel.
Your schema and the model’s understanding of your schema diverge. You add a field to support a new feature, forget to update the schema description, and the model starts guessing at what that field means based on its name alone. A field called status with no enum constraint and no description will get populated differently depending on training data, prompt phrasing, and which model you’re on.
Or you upgrade a model version and its interpretation of ambiguous instructions changes. Nothing about your schema changed, but the model’s default behavior for optional fields, null handling, or nested arrays did. You didn’t touch the code. The output shape still changed.
Or constrained decoding is silently unavailable for the path you’re using, which is the one that bites hardest. You built and tested against strict structured output, but a later prompt change routes you through a tool call path, a streaming path, or a model that doesn’t support strict mode for that particular schema shape (deeply nested objects, anyOf, recursive schemas, and very large enum lists are common places where strict mode support has gaps or extra restrictions). You fall back to prompted JSON without noticing, because the response still looks like JSON. It just isn’t guaranteed to match your schema anymore.
None of these require the model to be “wrong.” They’re integration failures that manifest as a KeyError three services downstream from where the actual generation happened.
Where it shows up in a real pipeline
If you’re running any kind of RAG pipeline or agent loop that chains structured output calls, the failure mode is a slow leak rather than a crash on the first call.
A common pattern: step one extracts entities into a schema with a confidence field typed as a float. Step two consumes that output and does something conditional on confidence > 0.7. Somewhere along the way, a model update or a prompt tweak causes the field to occasionally come back as the string "high" instead of a float, because the schema’s field description said “confidence level” without pinning down the type strictly enough, and the model pattern matched against training data where confidence was expressed as a category.
Your Python or TypeScript type hints don’t save you here, because they’re compile time hints for your code, not runtime validation of what the API actually returned. You need a runtime validator, Pydantic on the Python side or Zod on the TypeScript side, sitting directly between the API response and anything that touches that data, because the contract between “what your schema says” and “what came back” is enforced by nothing except that validator you wrote.
The retry tax nobody puts in the demo
When validation fails, you have three options: retry the call, patch the malformed output in code, or drop the record. Each has a cost that’s easy to underestimate when you’re prototyping against a handful of examples.
A retry means another full round trip. Depending on your prompt length and the model, that’s another few hundred to a few thousand input tokens billed again, plus the output tokens, plus the latency of a second generation on top of the first. If your pipeline processes any real volume, a retry rate that looks trivial at the “let me test this manually” stage, say a couple of failures out of twenty tries, turns into a meaningful chunk of your monthly API bill and a meaningful tail latency problem once you’re running thousands of calls a day. I don’t have a clean number to give you here because it depends entirely on your schema complexity and which model and mode you’re running, but the direction is consistent: retries are not free, and they scale with volume in a way that manual testing hides from you.
Patching in code, writing a small normalizer that catches the known drift patterns like "high" where a float was expected, works until the drift pattern changes again, at which point you’re maintaining a growing pile of special cases that exist only because you don’t have a hard guarantee from the API layer.
Dropping the record silently is the option nobody admits to shipping, and it’s also the most common one in codebases I’ve inherited. It looks fine in the logs because there’s no error. It just means your downstream data is quietly incomplete.
What actually helps
Pin field types tightly and use enums wherever the value space is closed. “Confidence” as a free-floating description is an invitation for drift. “Confidence” typed as a number with an explicit min and max, described in one sentence with no ambiguity, gives constrained decoding something concrete to hold onto and gives your validator something concrete to check.
Version your schemas the same way you’d version an internal API. If you change a field’s type or add a required field, that’s a breaking change to anything consuming the output, including your own downstream code that you wrote six months ago and forgot about.
Log the raw response alongside the validated one, at least for a sampling of production traffic. When drift happens, you want to see exactly what came back, not just that validation failed. This is the single most useful thing I’ve added to any pipeline that depends on structured output, because it turns “something’s wrong somewhere” into “field X started coming back as a string on Tuesday.”
Treat strict mode and prompted JSON as genuinely different reliability tiers, not interchangeable settings. If your use case can tolerate occasional malformed output behind a validator and retry loop, prompted JSON with a good schema description is fine and often cheaper. If a malformed field breaks something downstream that a human doesn’t see until it’s already caused damage, pay for the more constrained path where it’s available and confirm it’s actually engaged for the exact call shape you’re using, not just the shape you tested with.
The honest summary
Structured output narrows the gap between “the model said something” and “my code can use it,” but it doesn’t close that gap. It moves the risk from parsing free text to trusting an API guarantee that varies by provider, by model version, and by how deeply nested your schema is. The engineers who get burned by this are the ones who stopped validating once the demo worked, not the ones who skipped structured output.
If you’re building anything that chains structured calls together, budget for the validator, budget for the retry, and check on a regular cadence that the mode you think you’re running is the mode you’re actually getting.
For more breakdowns like this on what actually holds up in production versus what looks good in a demo, head back to the AI Tool Gazette homepage.