Skip to content

Merge streaming tool call deltas by index in OpenAiChatModel#6381

Open
jewoodev wants to merge 2 commits into
spring-projects:mainfrom
jewoodev:fix-stream-toolcall-merge-by-index
Open

Merge streaming tool call deltas by index in OpenAiChatModel#6381
jewoodev wants to merge 2 commits into
spring-projects:mainfrom
jewoodev:fix-stream-toolcall-merge-by-index

Conversation

@jewoodev

@jewoodev jewoodev commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

OpenAI-compatible streaming endpoints can send continuation tool-call deltas with id: "" instead of omitting id. OpenAiChatModel.ChunkMerger treats any present id as a new tool-call boundary, so argument fragments become separate ToolCalls with empty id/name and tool execution receives empty or null arguments.

This PR merges tool-call deltas by the required index field instead. It fixes the empty-id case from gh-6374, keeps OpenAI-style omitted ids working, and removes the one-tool-call-per-chunk assumption so parallel or multi-delta chunks merge correctly. #6369 covers the empty-id continuation case; this PR uses the index-keyed path so those related cases stay covered together.

Tests cover omitted ids, empty ids, parallel tool calls, and multiple deltas per chunk. Verified with ./mvnw -pl models/spring-ai-openai package.

Fixes gh-6374

@jewoodev jewoodev force-pushed the fix-stream-toolcall-merge-by-index branch from d1d69a0 to 6c04458 Compare June 11, 2026 03:42
@jewoodev

jewoodev commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main to resolve a conflict with #6365.

#6365 changed mergeDeltas to preserve tool-call additional properties. This PR replaces that merge logic with by-index merging, so I carried the preservation over: ChunkMerger.mergeToolCalls now copies each delta's additional properties via putAllAdditionalProperties.

The streaming test added in #6365 (preserveUnmappedToolCallAdditionalProperties) verifies this. It fails without that call and passes with it, along with the rest of the chat model tests.

The streaming chunk aggregation detects the start of a new tool call by
the presence of the delta id field. OpenAI omits id on continuation
deltas, but some OpenAI-compatible endpoints (e.g. DeepSeek) send an
empty string instead, so every continuation delta surfaced as a
separate tool call with fragmented arguments and tool execution failed.

Key the merge on the delta's required index field instead, which the
streaming API defines for correlating tool call deltas and which the
aggregation used until 2.0.0-M8. This also lifts the one-tool-call-per-
chunk limitation and merges parallel tool call deltas correctly.

Fixes spring-projectsgh-6374

Signed-off-by: jewoodev <jewoos15@naver.com>
@jewoodev jewoodev force-pushed the fix-stream-toolcall-merge-by-index branch from 6c04458 to b03dc62 Compare June 26, 2026 11:38
@freshman-wang

Copy link
Copy Markdown

Thanks for this PR — the index-based merging approach is much cleaner than my earlier attempt.

While testing with DashScope/Qwen, I also hit two NoSuchElementException cases in the same ChunkMerger that this PR does not yet cover. It would be great if they could be addressed here (or in a follow-up) so the streaming path is fully robust:

1. mergeDeltaslastFromTc1.function().get()

When a continuation chunk carries arguments only (no function object), lastFromTc1.function() is empty and .get() throws:

// current
Function lastFromTc1F = lastFromTc1.function().get();

Suggested:

Function lastFromTc1F = lastFromTc1.function()
    .orElseGet(() -> Function.builder().build());

2. chunkToChatCompletiontc.id().get(), tc.function().get().name().get(), tc.function().get().arguments().get()

All four .get() calls can throw when the corresponding fields are absent in the delta:

// current
toolCallBuilder.id(tc.id().get());
toolCallBuilder.function(ChatCompletionMessageFunctionToolCall.Function.builder()
    .name(tc.function().get().name().get())
    .arguments(tc.function().get().arguments().get())
    .build());

Suggested:

toolCallBuilder.id(tc.id().orElse(""));
ChatCompletionMessageFunctionToolCall.Function.Builder funcBuilder = ChatCompletionMessageFunctionToolCall.Function.builder();
tc.function().ifPresent(f -> {
    funcBuilder.name(f.name().orElse(""));
    funcBuilder.arguments(f.arguments().orElse(""));
});
toolCallBuilder.function(funcBuilder.build());

Reproduced on Spring AI 2.0.0 GA with DashScope Qwen3 streaming + tool_calls. Happy to open a separate issue if that would help track it. Thanks!

(Originally submitted as #6522, closed in favor of this PR and #6373.)

@freshman-wang

Copy link
Copy Markdown

@jewoodev Thanks for the great work on this PR!

Would you be open to me adding the NoSuchElementException fixes directly to this branch as a follow-up commit? That way the reviewer can see all the ChunkMerger fixes together.

If yes, could you add me as a collaborator (Write permission) to your fork at https://github.com/jewoodev/spring-ai/settings/access ? My GitHub username is freshman-wang.

If you prefer to keep the PR scope tight, I'm happy to wait and open a follow-up PR after this one merges. Just let me know which you prefer. Thanks!

Avoid direct Optional.get() calls when converting streamed tool call
deltas into chat completions. Missing tool call ids, functions, names,
and arguments are now normalized to empty strings during conversion.

Add regression coverage for arguments-only tool call deltas and deltas
without a function object.

Signed-off-by: jewoodev <jewoos15@naver.com>
@jewoodev

jewoodev commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback.

Added a small follow-up commit that guards the remaining chunkToChatCompletion Optional.get() calls and covers both arguments-only tool call deltas and deltas without a function object.

The mergeDeltas case is already covered by the by-index merge in this branch, since mergeToolCalls now builds a function when the previous delta does not carry one.

I'll keep follow-up changes on this branch on my side to keep the PR history straightforward. Feel free to leave any additional feedback in this PR.

@freshman-wang

Copy link
Copy Markdown

@sdeleuze Could you take a look at this PR when you have a chance? It fixes streaming tool-call merging for empty id continuations from OpenAI-compatible endpoints. The index-keyed approach handles parallel tool calls and multi-delta chunks correctly. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

3 participants