fix(gemini): surface model-generated images instead of dropping them - #41468
fix(gemini): surface model-generated images instead of dropping them#41468IngTian wants to merge 1 commit into
Conversation
Gemini image models (gemini-*-image, "Nano Banana") return generated
images as `inlineData` parts. The Gemini stream decoder only handled
`text` and `functionCall`, so image parts fell through the loop and were
discarded. The user is billed for the image output tokens and sees only
the model's text ("Here is the image you asked for") with no image.
There was no `file` event in the LLM event union to carry the media, so
this adds one and threads it through:
- `LLMEvent.File` — a complete media block. Media arrives whole rather
than as deltas, so there is no start/delta/end triple.
- Gemini decoder emits it for `inlineData`, closing any open reasoning
block first so media does not land inside it.
- `SessionProcessor` persists it as a `file` part, closing the current
text block first to preserve ordering, and reuses the same image
normalization tool attachments use so an oversized generated image
cannot blow up the next request. If normalization is unavailable it
keeps the original rather than dropping the image.
- `toModelMessages` replays assistant `file` parts, and the Gemini
request builder now accepts assistant media, so follow-up turns
("make it brighter") can edit a generated image instead of starting
over. Under `stripMedia` (compaction/overflow) it degrades to a text
placeholder like other media.
Note that `responseModalities: ["TEXT", "IMAGE"]` must still be set in
the model's provider options for Gemini to return images at all; without
it the API returns text only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Two notes for whoever picks this up. Relationship to #40126. That PR fixes the same gap on Pairs with #41463. On |
|
The following comment was made by an LLM, it may be inaccurate: I found a potentially related PR: PR #40126: feat(session): support Gemini image generation This PR appears to be related as it also addresses Gemini image generation support. You should check whether this earlier PR covers similar ground or if there's any overlap in approach between the two PRs. |
The image-drop bug now has a dev-targeted PR (anomalyco/opencode#41468, issue #41467) rather than only the v2-branch one, so both halves of the upstream fix are traceable from here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issue for this PR
Closes #41467
Type of change
What does this PR do?
Gemini image models return generated images as
inlineData, but nothing carries them into the session, so the image is dropped and the user sees only the model's text. The request is still billed.Two gaps, so three small changes:
LLMEvent.File— there was no event to carry media. Media arrives whole rather than as deltas, so there's no start/delta/end triple.inlineData, closing any open reasoning block first so media doesn't land inside one.SessionProcessorpersists it as afilepart, closing the current text block first to keep ordering. It reuses the same image normalization tool attachments already use, so an oversized generated image can't blow up the next request; if normalization is unavailable it keeps the original rather than dropping the image.Also:
toModelMessagesreplays assistantfileparts and the Gemini request builder accepts assistant media, so a follow-up turn ("make it brighter") can edit the generated image instead of starting over. UnderstripMedia(compaction/overflow) it degrades to a text placeholder like other media.Note
responseModalities: ["TEXT", "IMAGE"]must still be set in the model's provider options, or Vertex returns text only.How did you verify your code works?
Tests:
packages/llm299 pass,packages/opencodesession suite 399 pass / 0 fail,typecheckclean on both.New coverage — a Gemini decoder test asserting a
fileevent is emitted forinlineDataalongside the text, a processor test asserting the image is persisted as afilepart, and twotoModelMessagestests for replay andstripMedia. I checked the processor test fails without the fix, so it's real coverage rather than a tautology.I also updated the existing
rejects unsupported assistant media contenttest, which asserted the old behavior ("only support text, reasoning, and tool-call content for now") — that's the placeholder this PR fills in.Not verified: live Vertex traffic. Decoder behavior is covered by a synthetic SSE fixture rather than a real provider response.
Screenshots / recordings
Not a UI change — this makes the
filepart exist; existing clients render it.Checklist