feat(agent): validate structured outputs after generation with one retry

[mzxchandra]

Problem

The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _responseFormatWarning that nothing in the repo reads, and returns success via the standard-format fallback. Downstream blocks read the structured fields as undefined, indistinguishable from a legitimate empty answer.

Two enforcement gaps make this worse:

• Native structured outputs are validated against a weakened schema. For models with nativeStructuredOutputs, the Anthropic provider passes the schema through the SDK's transformJSONSchema, which strips enum, const, minLength/maxLength, pattern, numeric minimum/maximum, and maxItems out of the enforced grammar and pastes them into description as advisory prose. Outputs are grammar-valid but can violate the authored constraints.
• Prompt-based models have zero enforcement — the schema is system-prompt text only, and nothing checked the result.

Observed in production-shaped runs (~130 extraction workflows): enum-violating hollow rows emitted mid-degeneration, importance: 0 against a prose-only "1 to 10" bound, and 128k max_tokens blowups whose truncated JSON failed parse and silently became success.

Change

A new executor/handlers/agent/structured-output.ts validates every completed (non-streaming) structured response in the provider-request seam of the agent handler, treating three failure modes as one failure condition:

1. Truncation — the final model trace segment's finishReason is max_tokens / length / MAX_TOKENS (case-insensitive). Every provider's trace enricher already records this on the model time segments, so no per-provider wiring is needed.
2. Unparseable JSON — with a Markdown code-fence rescue before failing, since prompt-based models commonly fence otherwise-valid JSON.
3. Schema violation — ajv (already a dependency; strict: false so user-authored schemas with unknown keywords still compile) validates against the authored schema, restoring enum, minLength, and numeric bounds to enforced status on both native and prompt-based paths. A schema ajv cannot compile degrades to the parse-only checks rather than failing runs.

On failure: one identical resample, but only for requests without tools — tools execute inside the provider call, so a resample would replay their side effects. The failed attempt's tokens, cost, and trace segments are folded into the final result so the block's reported usage matches actual spend. A second failure (or the first, on a tool-carrying request) fails the block explicitly with the validation reason.

Where the knob lives

No new settings. Enforcement follows the response format's existing strict flag, which the block's input schema has always documented as defaulting to true and which parseResponseFormat already normalizes onto every format — it was previously read by nothing. "strict": false keeps the exact legacy lenient fallback (including the _responseFormatWarning).

Deliberately not model-conditional at the surface: the UI's model value can be an unresolved <block.output> reference or sim-auto (resolved only at execution), so a model-conditional field cannot know the actual model — and prompt-based models need validation strictly more. Deliberately not the block-level retry machinery: that wraps the entire handler and would replay memory reads, MCP discovery, and tool calls.

Streaming responses are unchanged — there is no complete content to validate at this seam.

Side benefit for schema authors

With post-generation validation in place, minLength: 1, minimum/maximum, and maxItems in response format schemas become meaningful for the first time on every model. (Also worth knowing: format: "date" is in the SDK's supported-formats set, so date fields get native grammar enforcement just by declaring it.)

Testing

• New structured-output.test.ts (17 tests): fixtures modeled on the observed corruption — hollow enum-violating rows, truncated JSON, fenced JSON, MAX_TOKENS/length finish reasons, uncompilable schemas, bare-schema formats, usage merging.
• Handler-level tests: retry-then-succeed with token merging and identical resample request, fail-after-retry, schema-violation retry, strict: false legacy fallback, no-resample-with-tools, max_tokens tripwire.
• Six existing provenance tests updated to return schema-valid mock content (their 'Mocked response content' default now correctly fails strict validation).
• bun run type-check, bun run check:api-validation, bun run check:boundaries, and the full agent-handler suite (115 tests) pass.

Docs: the agent block's Response Format section now describes enforcement, the retry, and the strict: false opt-out.

🤖 Generated with Claude Code

https://claude.ai/code/session_018L7CosQfKB9SGCF8E9fLfY

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 6, 2026 1:19am UTC

[mzxchandra]

Field data behind the retry design, from ~130 production-shaped extraction runs measured the night this was built:

• Per-call corruption rate was ~10-14% on that workload (parse failures, hollow enum-violating rows, cap truncations). One identical resample takes the residual failure rate to ~1-2%, which is why the PR ships a single retry rather than a configurable count — the expected-value math flattens out fast after one.
• Not all failures are stochastic. One real input document corrupted 8 of 10 attempts — content-triggered degeneration, not sampling noise. Resampling cannot make that class vanish; the explicit block failure is the mechanism that surfaces it (previously those runs reported success with undefined fields).

So reviewers should expect this change to (a) silently fix roughly nine in ten of the failures it detects via the resample, and (b) newly fail the small remainder that were previously invisible successes. Both are intended.

[greptile-apps]

Greptile Summary

This PR adds post-generation validation for strict Agent structured outputs, one tool-safe resampling attempt, accumulated retry usage metadata, regression tests, and corresponding documentation.

• Detects token-limit truncation, malformed JSON, and authored-schema violations.
• Preserves the legacy fallback when strict is false and avoids retrying requests that carry tools.
• The validation path still needs format-keyword enforcement and should avoid collecting unbounded validation errors.

Confidence Score: 4/5

The PR should not merge until strict validation actually enforces supported JSON Schema formats; the unbounded error collection is an additional non-blocking performance concern.

The retry and core JSON/schema paths are well covered, but the Ajv instance silently accepts values that violate authored format constraints, leaving a concrete correctness gap in the feature’s strict-validation contract.

Files Needing Attention: apps/sim/executor/handlers/agent/structured-output.ts

Important Files Changed

Filename	Overview
apps/sim/executor/handlers/agent/structured-output.ts	Adds parsing, truncation detection, schema validation, validator caching, and retry-usage merging; format assertions are currently ignored and all validation errors are unnecessarily collected.
apps/sim/executor/handlers/agent/agent-handler.ts	Integrates strict validation and a single tool-safe retry at the provider-request seam while preserving streaming and lenient paths.
apps/sim/executor/handlers/agent/structured-output.test.ts	Covers parsing, authored constraints, truncation vocabularies, schema compilation fallback, and usage merging, but lacks a format-keyword case.
apps/sim/executor/handlers/agent/agent-handler.test.ts	Adds handler-level coverage for retry success and failure, schema violations, strict opt-out, tools, truncation, and token merging.
apps/docs/content/docs/workflows/blocks/agent.mdx	Documents post-generation enforcement, retry behavior, tool-bearing request behavior, explicit failure, and the strict opt-out.

Sequence Diagram

sequenceDiagram
  participant A as Agent handler
  participant P as Provider
  participant V as Structured-output validator
  A->>P: Execute request
  P-->>A: Completed response
  A->>V: Check finish reason, parse JSON, validate schema
  alt Valid
    V-->>A: Parsed structured output
    A-->>A: Return output and metadata
  else Invalid and tool-free first attempt
    V-->>A: Validation reason
    A->>P: Resample identical request once
    P-->>A: Second response
    A->>V: Validate again
  else Invalid with tools or after retry
    V-->>A: Validation reason
    A-->>A: Fail block explicitly
  end

Loading

_{Reviews (1): Last reviewed commit: "feat(agent): validate structured outputs..." | Re-trigger Greptile}

[greptile-apps]

Format constraints are ignored

Schemas using format are not fully enforced because this Ajv instance has no format implementations registered. For example, a prompt-based model can return {"date":"not-a-date"} for a property declared with format: "date", and the strict validator will accept it. Downstream blocks can therefore receive values that violate the authored schema. Register the supported formats or reject unsupported format keywords instead of silently ignoring them.

[greptile-apps]

Validation collects excess errors

Enabling allErrors makes Ajv allocate an error object for every violation in potentially large model outputs, although only five errors are displayed. A large generated array with many invalid elements can therefore create avoidable CPU and memory pressure during executor runs. Consider using bounded or first-error validation.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!