unstable/ai: recover unknown tool calls - #7574
Open
candrewlee14 wants to merge 1 commit into
Open
Conversation
🦋 Changeset detectedLatest commit: 1b9f096 The changes in this PR will be included in the next version bump. This PR includes changesets to release 30 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 2, 2026 01:11
6f18e2e to
2632a0e
Compare
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 2, 2026 01:34
2632a0e to
63e3da1
Compare
candrewlee14
changed the title
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 2, 2026 04:37
63e3da1 to
c62edc4
Compare
candrewlee14
changed the title
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 2, 2026 04:47
c62edc4 to
6306aeb
Compare
effect-slopcop
Bot
added
enhancement
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 2, 2026 04:53
6306aeb to
d3bbf0b
Compare
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 2, 2026 05:29
ae1bab7 to
1a53a2f
Compare
effect-slopcop
Bot
added
bug
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 2, 2026 05:46
1a53a2f to
5267d26
Compare
candrewlee14
changed the title
effect-slopcop
Bot
added
bug
Contributor
|
Hey! It will be a while until this gets a proper review as it is a large change and we have higher priorities atm :) |
Contributor
Author
|
Thanks for letting me know! I'm also totally understanding if you throw this out and solve it in a different way, of course. |
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 5, 2026 04:59
5267d26 to
18e7a62
Compare
candrewlee14
changed the title
A tool call naming a tool which is not in the toolkit fails the whole operation today: `Response.Part(toolkit)` has no member for it, and even if it decoded, `resolveToolCalls` would skip it and leave it unanswered, which the next request rejects. For an agent whose toolkit changes between turns that is an ordinary event rather than a hallucination. `generateText` and `streamText` accept `unknownToolCalls`. With `"return"` such a call comes back as a `tool-call-error` part carrying the original call, its JSON parameters, and a `ToolNotFoundError`, and `Prompt.fromResponseParts` adds it to history as the call plus a failed tool result so the model can correct it. The default is unchanged, and the part is absent from the response type unless the option is set. An operation with no toolkit, or an empty one, treats every tool call as unknown, so the option applies there too. A call which names a tool in the toolkit is untouched: `Toolkit` routes a failure of its parameters through that tool's `failureMode`, and with resolution disabled it fails the operation as before. Recovery is driven by the decode which already validates the response: parts are decoded one at a time, and a failure is recoverable only when the part is a call to a tool which is not in the toolkit. Parameters are never validated twice, and any other decode failure fails the operation as before. `HttpRequestDetails` and `HttpResponseDetails` move to a leaf module so `Response` can reference `AiError` without a cycle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019wFhhXHNc4Fnkye1orWQqr
candrewlee14
force-pushed
the
fix/tool-call-recovery-opt-in
branch
from
September 6, 2026 15:26
18e7a62 to
1b9f096
Compare
candrewlee14
changed the title
danieljvdm
suggested changes
Sep 10, 2026
Contributor
There was a problem hiding this comment.
the new response part and generic make sense for keeping the tool types intact. my concern is recovery rewriting calls while execution still reads the originals.
could we classify each call once and use that for recovery, execution and history? we'd still keep the raw params for Toolkit to decode.
Comment on lines
+637
to
+640
| export type ExtractUnknownToolCalls<Options> = Options extends { | ||
| readonly unknownToolCalls: infer Mode | ||
| } ? [Exclude<Mode, undefined>] extends [never] ? "error" : Extract<Mode, "error" | "return"> | ||
| : "error" |
Contributor
There was a problem hiding this comment.
optional unknownToolCalls gets typed as "error" even when set to "return", so toolCallErrors becomes never[]. same issue in streaming and UnknownToolCallsOf.
| // `Toolkit` routes its parameter failure through the tool's own | ||
| // `failureMode` when it resolves the call. When resolution is disabled | ||
| // the caller owns the call, so it fails here exactly as it does today. | ||
| if (Object.hasOwn(toolkit.tools, part.name)) { |
Contributor
There was a problem hiding this comment.
toString, constructor and __proto__ still fail in return mode. the remaining lookups need Object.hasOwn too.
toString with finish reason "length" also produces two results for one call.
Comment on lines
+2554
to
+2560
| decodePart({ | ||
| type: "tool-call-error", | ||
| id: part.id, | ||
| name: part.name, | ||
| params: toJson(part.params), | ||
| error: encodeAiError(AiError.make({ module: "LanguageModel", method, reason })), | ||
| metadata: part.metadata |
Contributor
There was a problem hiding this comment.
providerExecuted: "true" gets dropped here and escapes validation. same for 1 and null. can we validate before converting?
Comment on lines
+2402
to
+2411
| for (const part of result.success.content) { | ||
| if (part.type === "tool-call-error") { | ||
| strictEqual(typeof part.id, "string") | ||
| strictEqual(typeof part.name, "string") | ||
| assertTrue(AiError.isAiError(part.error)) | ||
| } | ||
| if (part.type === "tool-call") { | ||
| strictEqual(typeof part.id, "string") | ||
| strictEqual(typeof part.name, "string") | ||
| } |
Contributor
There was a problem hiding this comment.
can we assert the full schema here and generate invalid providerExecuted values too?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #7452, rewritten on top of #7588. Supersedes #6432.
Problem
If the model calls a tool that isn't in the toolkit, the whole
generateText/streamTextcall fails.Response.Part(toolkit)builds its union from the toolkit's tools, so there's no schema member for the unknown name and the decode fails. Even if it decoded,resolveToolCallsskips it, so it'd be left without a result and the next request would be rejected.For agents whose toolkit changes between turns this is a normal thing to happen, not a hallucination. The model saw the tool a turn ago and now it's gone. Telling the model "that tool doesn't exist, try again" is a better outcome than failing the turn.
Change
generateTextandstreamTexttake a new option,unknownToolCalls: "error" | "return". Default is"error", which is today's behaviour. With"return", an unknown tool call comes back as atool-call-errorpart with the original call, its params, and aToolNotFoundError.Prompt.fromResponsePartsturns that into the assistant tool call plus a failed tool result, so the model can correct itself.Calls to tools that are in the toolkit aren't affected. #7588 already routes their parameter failures through
failureMode, and I left that alone. WithdisableToolCallResolution: truethose calls still fail the operation like they do on main. So the split is: the tool decides for its own calls, the operation decides for calls that don't belong to any tool. If there's no toolkit, or it's empty, every call is unknown and the option applies the same way.On @IMax153's concern from #7452 about callers silently missing invalid calls: with the default,
toolCallErrorsisArray<never>and"tool-call-error"isn't in the response part type, so nothing changes for existing code unless you opt in.Notes
tool-callnaming an unknown tool is recovered. Anything else that fails to decode still fails the operation. Params are never validated twice.tool-call-errorpart is produced by decoding, not constructed by hand, so a call that's malformed in some other way (non-stringid, bad metadata) keeps failing rather than coming back under a misleading error.Formatterso they're JSON-safe in history (the1ncase from the last review).HttpRequestDetails/HttpResponseDetailsmoved tointernal/http-details.tssoResponsecan importAiErrorwithout a cycle. Both modules still re-export them.Response.AllPartsincludes the new part unconditionally.Tests cover
generateText,streamTextandgenerateObject, with a toolkit, without one, and with an empty one, resolution on and off, a non-literal option value, and a property test that a returned part always satisfies the response schema. Also asserted: known-tool parameter failures still fail with resolution disabled, an unrecoverable response runs no handlers, and history never ends up with a call without a result or a result without a call.If you'd rather this lived on the toolkit (a second type parameter, which would flow to
ChatandgenerateObjectfor free) I'm happy to rework it that way.🤖 Generated with Claude Code
https://claude.ai/code/session_019wFhhXHNc4Fnkye1orWQqr