Skip to content

feat(session): support Gemini image generation - #40126

Open
Eric-Guo wants to merge 9 commits into
anomalyco:v2from
Eric-Guo:t_rax
Open

feat(session): support Gemini image generation#40126
Eric-Guo wants to merge 9 commits into
anomalyco:v2from
Eric-Guo:t_rax

Conversation

@Eric-Guo

@Eric-Guo Eric-Guo commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Issue for this PR

Closes #40124

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Gemini can return generated images as inline data, but V2 did not carry that output through the session pipeline. This PR emits durable assistant file events, projects and renders generated images, preserves Gemini thought signatures for replay and follow-up edits, and handles filesystem change invalidation for generated files.

How did you verify your code works?

  • bun turbo typecheck --concurrency=3 (34 packages)
  • AI Gemini/schema tests (34 passing)
  • Core session runner tests (30 passing)
  • App reducer and file watcher tests (9 passing)
  • Regenerated the client from the public schema

Screenshots / recordings

WeChatWorkScreenshot_d4bd283c-8e89-449c-b6b6-69a0a1bbb997

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The TUI transcript formatter currently embeds data: URLs directly into Markdown output for file parts, which can explode transcript size and degrade performance/UX.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Adds end-to-end support for Gemini-generated inline images in the V2 session pipeline by persisting them as durable assistant “file” parts, projecting them into session state, and rendering them across UIs and transcripts (with provider metadata preserved for replay/edit flows).

Changes:

  • Introduces SessionMessage.AssistantFile content + durable session.file.generated event, and projects it into session state.
  • Publishes file events from LLM streaming (Gemini inlineData) and restores provider metadata when replaying assistant media.
  • Updates TUI, session-ui, app reducer, and CLI/mini transports to recognize the new file part type.
File summaries
File Description
packages/tui/src/routes/session/rows.ts Adds session.file.generated row handling and resolves file parts by id.
packages/tui/src/routes/session/index.tsx Renders generated file parts in TUI and includes them in transcript formatting.
packages/tui/src/mini/stream-v2.transport.ts Skips file parts in mini transport tool rendering loop.
packages/tui/src/mini/stream-v2.subagent.ts Skips file parts when tracking subagent tool events.
packages/session-ui/src/components/message-part.tsx Adds UI rendering for assistant file parts with image preview support.
packages/session-ui/src/components/message-part.css Adds styling for the new file/image part rendering.
packages/schema/src/session-message.ts Adds AssistantFile schema/type and includes it in assistant content union.
packages/schema/src/session-event.ts Adds durable session.file.generated event definition and inventory entry.
packages/core/test/session-runner-tool-events.test.ts Adds coverage ensuring model-generated files are emitted as durable assistant file content.
packages/core/test/session-runner-message.test.ts Adds coverage for restoring provider metadata for generated assistant media/files.
packages/core/src/session/runner/to-llm-message.ts Maps assistant file parts back into LLM “media” messages (with provider metadata).
packages/core/src/session/runner/publish-llm-event.ts Publishes session.file.generated from LLM file events and generates filenames/data URLs.
packages/core/src/session/projector.ts Projects session.file.generated durable events into the session projector.
packages/core/src/session/message-updater.ts Updates assistant messages by appending generated file parts on session.file.generated.
packages/core/src/session/compaction.ts Includes file parts in compaction serialization (summary line).
packages/client/src/promise/generated/types.ts Regenerates client types to include assistant file content + new durable event type.
packages/client/src/effect/api/api.ts Regenerates Effect client API union to include session.file.generated.
packages/cli/src/run/noninteractive.ts Avoids treating file parts as tools during noninteractive rendering.
packages/cli/src/acp/event.ts Replays assistant file parts into ACP session updates.
packages/app/src/utils/session-message.ts Projects assistant file parts into app Part model for rendering/state.
packages/app/src/context/server-session-v2-reducer.ts Projects session.file.generated into live assistant content (deduping by file id).
packages/app/src/context/server-session-v2-reducer.test.ts Adds reducer test ensuring generated files appear in assistant content and are deduped.
packages/app/src/context/file/watcher.ts Extends watcher invalidation to handle filesystem.changed events.
packages/app/src/context/file/watcher.test.ts Tests invalidation behavior for both legacy and current watcher event types.
packages/ai/test/schema.test.ts Adds decoding coverage for LLM file events.
packages/ai/test/provider/gemini.test.ts Adds Gemini tests for emitted file events and replaying assistant media with thought signatures.
packages/ai/src/schema/messages.ts Adds optional providerMetadata to media parts.
packages/ai/src/schema/events.ts Adds LLM file event schema + reducer support to append media content.
packages/ai/src/protocols/gemini.ts Adds Gemini lowering/parsing support for assistant media and inlineData→file event emission.
Review details
  • Files reviewed: 28/29 changed files
  • Comments generated: 1
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread packages/tui/src/routes/session/index.tsx
@Eric-Guo

Eric-Guo commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

The TUI transcript formatter currently embeds data: URLs directly into Markdown output for file parts, which can explode transcript size and degrade performance/UX.

Data URLs now render as a short filename label; normal URLs remain links.

Root cause: commit 7cd9ffa ("Preserve Gemini thought signatures for text and images") added a file variant to AssistantContent in packages/schema/src/session-message.ts:193, but transfer.ts's sanitize code assumed anything that isn't text/reasoning is a tool — so it tried to read providerState/providerResultState/ToolState off the file variant.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants