Structured outputs vs asking nicely for JSON: why one is reliable and the other isn't

2 min read

"Respond only in valid JSON matching this schema" in a prompt works most of the time. Most of the time is not the same as reliably, and the failure mode — a response that's almost valid JSON, with a trailing comma or an extra sentence before the opening brace — tends to show up in production, not in testing.

Why prompted JSON fails intermittently

Asking nicely is a request the model can decline to fully honor, not a constraint it's mechanically unable to violate. A model asked to "respond only in JSON" might still add a preface sentence, wrap the JSON in a markdown code fence, use a field name close to but not exactly what was specified, or produce technically-invalid JSON (unescaped quotes, trailing commas) under some phrasings of the input. None of these are common failures, which is exactly the problem — a pipeline built on prompted JSON works fine through testing and then breaks on a production input nobody tried, at a rate too low to catch reliably before shipping and too high to treat as acceptable for anything downstream that does json.loads() on the raw response.

What structured outputs actually change

A structured-outputs feature constrains the response format at the API level, not just through prompt wording — the response is validated against a provided JSON Schema as part of generation, not requested and hoped for. Concretely, this usually means passing a schema as a request parameter rather than describing it in prose:

# Prompted (unreliable): describe the shape in words, hope it's followed
response = client.messages.create(
    model="claude-opus-4-8",
    messages=[{"role": "user", "content": "Extract name and email as JSON: ..."}],
)
# response might be JSON, might have a preamble sentence, might not parse

# Structured (reliable): the schema is enforced, not requested
response = client.messages.create(
    model="claude-opus-4-8",
    messages=[{"role": "user", "content": "Extract name and email: ..."}],
    output_config={"format": {"type": "json_schema", "schema": {
        "type": "object",
        "properties": {"name": {"type": "string"}, "email": {"type": "string"}},
        "required": ["name", "email"],
        "additionalProperties": False,
    }}},
)
# response is guaranteed to match the schema shape

The difference isn't "better prompting produces the same guarantee" — it's a different mechanism entirely. Prompting influences what the model is likely to produce; structured outputs constrain what a valid response is allowed to look like at all.

When prompted JSON is still fine

For a low-stakes internal tool, a one-off script, or a case where a human reviews the output before anything consumes it programmatically, the occasional malformed response might genuinely not matter enough to justify the extra setup. The distinction matters most specifically when something downstream calls a JSON parser on the raw response with no human in the loop — that's the exact case where an intermittent, hard-to-reproduce failure turns into a production incident instead of a caught bug.

Sources: Anthropic's Claude API documentation on structured outputs (output_config.format, JSON Schema validation, and the distinction from prompt-requested formatting).