Skip to content

Add image-generation hosted tool to the OpenAI Responses provider - #631

Open
PratikDhanave (PratikDhanave) wants to merge 4 commits into
microsoft:mainfrom
PratikDhanaveFork:image-generation-hosted-tool
Open

Add image-generation hosted tool to the OpenAI Responses provider#631
PratikDhanave (PratikDhanave) wants to merge 4 commits into
microsoft:mainfrom
PratikDhanaveFork:image-generation-hosted-tool

Conversation

@PratikDhanave

Copy link
Copy Markdown
Contributor

What

Adds hostedtool.ImageGeneration, a hosted-tool marker mirroring the existing WebSearch, FileSearch, CodeInterpreter and MCPServer types, and wires it into the OpenAI Responses provider so it is serialized into params.Tools as a responses.ToolImageGenerationParam.

Why

The Responses provider already handles the output side of image generation: responsesProcessResponse and the streaming path decode image_generation_call items into DataContent (covered by TestResponsesNonStreamingImageGenerationCall_MapsToDataContent). The underlying SDK also supports the request side via ToolUnionParam.OfImageGeneration. But the tool switch in responsesBuildCompletionParams handled only FuncTool, WebSearch, FileSearch, CodeInterpreter and MCPServer, so there was no way to actually request image generation through the hostedtool abstraction — params.Tools never got an image_generation tool.

This closes that input-side gap and brings the Go port in line with the .NET/Python HostedImageGenerationTool, keeping the hosted-tool surface consistent across SDKs (cross-SDK alignment).

How it works

  • ImageGeneration carries its optional settings (size, quality, background, output_format, partial_images) via an AdditionalProperties map, exactly like WebSearch/FileSearch.
  • The new case *hostedtool.ImageGeneration reads those keys, populates a responses.ToolImageGenerationParam, and appends a responses.ToolUnionParam{OfImageGeneration: ...}, following the existing WebSearch/CodeInterpreter cases.

Tests

TestResponsesImageGenerationTool_MapsToRequestParams runs through the existing fake-transport harness: it sends &hostedtool.ImageGeneration{AdditionalProperties: {size, quality}} and asserts the outgoing request body contains a image_generation tool with size/quality populated (the server-side body assertion is the black-box proxy for params.Tools), then confirms the response still maps back to DataContent. It fails before the switch case is added (the request omits the tools array) and passes after. Pairs with the existing TestResponsesNonStreamingImageGenerationCall_MapsToDataContent.

go build ./..., go vet and go test pass for the changed packages.

Open design questions

  • AdditionalProperties option surface: the settings are passed as an untyped map[string]any (size/quality/background/output_format/partial_images) to match the WebSearch/FileSearch precedent. Is a loosely-typed map the desired long-term shape, or would typed fields on ImageGeneration (mirroring FileSearch.MaximumResultCount) be preferred?
  • Scope: this PR maps size, quality, background, output_format and partial_images. The SDK also exposes model, moderation, input_fidelity, action and input_image_mask (inpainting). Happy to add those here or leave as follow-ups.
  • partial_images typing: currently accepts int/int64 from the map. If we keep the map surface, should JSON-number semantics (float64) also be accepted?

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions github-actions Bot added the public-api-change Pull Request changes public APIs label Jul 24, 2026
Introduce hostedtool.ImageGeneration mirroring the other hosted tool
markers, and map it in the Responses provider to a
responses.ToolImageGenerationParam. The Responses output path already
decodes image_generation_call items into DataContent, but there was no
way to request image generation through the hostedtool abstraction, so
params.Tools never carried an image_generation tool. This closes that
input-side gap and aligns with the .NET/Python HostedImageGenerationTool.

Size, quality, background, output_format and partial_images are carried
via AdditionalProperties, matching the WebSearch/FileSearch convention.
@github-actions

This comment has been minimized.

# Conflicts:
#	provider/openaiprovider/responses.go
#	tool/hostedtool/hostedtool.go
@github-actions

This comment has been minimized.

@PratikDhanave
PratikDhanave (PratikDhanave) marked this pull request as ready for review August 4, 2026 06:06
@PratikDhanave
PratikDhanave (PratikDhanave) requested a review from a team as a code owner August 4, 2026 06:06
Copilot AI lite review requested due to automatic review settings August 4, 2026 06:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class support for requesting image generation via the hosted-tool abstraction when using the OpenAI Responses provider, aligning the Go implementation with other SDKs and closing the “request-side” gap (the provider already handled image-generation outputs).

Changes:

  • Introduces hostedtool.ImageGeneration as a hosted-tool marker carrying image-generation options via AdditionalProperties.
  • Wires *hostedtool.ImageGeneration into responsesBuildCompletionParams so it serializes to responses.ToolUnionParam{OfImageGeneration: ...} (including mapped options like size, quality, etc.).
  • Adds a new black-box request/response test asserting the tool is present in the outgoing request body and that the response still maps back to DataContent.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
tool/hostedtool/hostedtool.go Adds the ImageGeneration hosted-tool marker and metadata to match the existing hosted tool patterns.
provider/openaiprovider/responses.go Extends the Responses tool-switch to serialize ImageGeneration into params.Tools with supported option mappings.
provider/openaiprovider/responses_test.go Adds a regression test verifying request-side tool serialization for image_generation and output mapping to DataContent.

@gdams

Copy link
Copy Markdown
Member

PratikDhanave (@PratikDhanave) can you resolve the parity gaps?

@github-actions github-actions Bot added area:provider Changes files in the provider area area:provider/openai Changes files in the provider / openai area area:tool Changes files in the tool area size:medium At most 100 changed lines across at most 5 files pending-auto-risk Automatic risk classification is in progress labels Aug 20, 2026
@github-actions

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by Go API Consistency Review Agent · sonnet46 · 139.9 AIC · ⌖ 4.96 AIC · ⊞ 6K

AdditionalProperties map[string]any
}

func (t *ImageGeneration) Name() string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-repo parity: model field is missing (required in upstream .NET)

The upstream .NET ResponseTool.CreateImageGenerationTool — and its FoundryAITool wrapper — takes model as a required parameter:

public static AITool CreateImageGenerationTool(
    string model,                           // required
    ImageGenerationToolQuality? quality = null,
    ImageGenerationToolSize? size = null,
    ImageGenerationToolOutputFileFormat? outputFileFormat = null,
    int? outputCompressionFactor = null,
    ImageGenerationToolModerationLevel? moderationLevel = null,
    ImageGenerationToolBackground? background = null,
    ImageGenerationToolInputFidelity? inputFidelity = null,
    ImageGenerationToolInputImageMask? inputImageMask = null,
    int? partialImageCount = null)

The Go ImageGeneration struct currently has no way to express a model, so callers that rely on a specific image model (e.g. gpt-image-1) cannot specify it through the hosted-tool abstraction. The OpenAI Responses API ToolImageGenerationParam exposes Model as an optional field on the request side, so Go could add a Model string field (populated when non-empty, following the FileSearch.MaximumResultCount precedent) without breaking the untyped map approach for the other settings.

The PR description notes model as a follow-up scope item. Flagging here so it is tracked as a parity gap before the type is considered stable. Happy to defer if the intent is to keep the surface minimal for now, but the gap should be explicit.

Other missing fields (output_compression_factor, moderation, input_fidelity, input_image_mask) are acknowledged in the PR description as explicit follow-ups — no separate comment needed for those.

@github-actions github-actions Bot added failed-auto-risk Automatic risk classification was inconclusive or failed and removed pending-auto-risk Automatic risk classification is in progress labels Aug 20, 2026
@github-actions github-actions Bot added pending-auto-risk Automatic risk classification is in progress risk:medium Contained production impact requiring normal review depth and removed failed-auto-risk Automatic risk classification was inconclusive or failed labels Aug 22, 2026
@github-actions github-actions Bot removed the pending-auto-risk Automatic risk classification is in progress label Aug 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Cross-Repo Parity Review

Verdict: ✅ Parity approved — minor documentation gap noted below

What was reviewed

PR adds hostedtool.ImageGeneration (a new exported type in tool/hostedtool) and wires it into the OpenAI Responses provider (provider/openaiprovider/responses.go). Both files are public, user-facing API.

Upstream equivalents found

SDK Equivalent surface
Python ImageGeneration TypedDict (imported from openai.types.responses.tool_param) + OpenAIChatClient.get_image_generation_tool() static method in python/packages/openai/agent_framework_openai/_chat_client.py
.NET No dedicated hosted-tool abstraction type found; CreateResponse.Tools accepts raw JsonElement[], so .NET callers build tool params directly. No HostedImageGenerationTool class exists in the .NET public API.

Contract comparison

The Go ImageGeneration.AdditionalProperties map handles size, quality, background, output_format, partial_images — all present in Python's get_image_generation_tool().

Python additionally exposes model, moderation, and output_compression parameters. These are valid fields in the underlying OpenAI SDK's ToolImageGenerationParam. The Go implementation does not map these from AdditionalProperties; a caller passing them would have them silently ignored. The PR itself flags this as an open question ("Scope"), so the omission is intentional.

Example coverage gap

Python ships client_image_generation.py and client_streaming_image_generation.py samples. Go has no equivalent in examples/. This is not a parity blocker, but a follow-up example would complete sample-parity coverage.

Label status

public-api-change is already present — correct, since hostedtool.ImageGeneration is a new exported type.


Adding parity-approved. The Go contract is semantically aligned with the Python upstream. The three unimplemented AdditionalProperties keys (model, moderation, output_compression) are noted in the PR's open questions and do not constitute a behavioral divergence from upstream defaults.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by Go API Consistency Review Agent · sonnet46 · 75.9 AIC · ⌖ 5.83 AIC · ⊞ 6K ·

@github-actions github-actions Bot added the parity-approved Go API consistency review found no parity issues label Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:provider/openai Changes files in the provider / openai area area:provider Changes files in the provider area area:tool Changes files in the tool area parity-approved Go API consistency review found no parity issues public-api-change Pull Request changes public APIs risk:medium Contained production impact requiring normal review depth size:medium At most 100 changed lines across at most 5 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants