Add image-generation hosted tool to the OpenAI Responses provider - #631
Add image-generation hosted tool to the OpenAI Responses provider#631PratikDhanave (PratikDhanave) wants to merge 4 commits into
Conversation
0147574 to
4ad3f58
Compare
This comment has been minimized.
This comment has been minimized.
4ad3f58 to
0a22730
Compare
This comment has been minimized.
This comment has been minimized.
Introduce hostedtool.ImageGeneration mirroring the other hosted tool markers, and map it in the Responses provider to a responses.ToolImageGenerationParam. The Responses output path already decodes image_generation_call items into DataContent, but there was no way to request image generation through the hostedtool abstraction, so params.Tools never carried an image_generation tool. This closes that input-side gap and aligns with the .NET/Python HostedImageGenerationTool. Size, quality, background, output_format and partial_images are carried via AdditionalProperties, matching the WebSearch/FileSearch convention.
0a22730 to
e89371c
Compare
This comment has been minimized.
This comment has been minimized.
# Conflicts: # provider/openaiprovider/responses.go # tool/hostedtool/hostedtool.go
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Pull request overview
Adds first-class support for requesting image generation via the hosted-tool abstraction when using the OpenAI Responses provider, aligning the Go implementation with other SDKs and closing the “request-side” gap (the provider already handled image-generation outputs).
Changes:
- Introduces
hostedtool.ImageGenerationas a hosted-tool marker carrying image-generation options viaAdditionalProperties. - Wires
*hostedtool.ImageGenerationintoresponsesBuildCompletionParamsso it serializes toresponses.ToolUnionParam{OfImageGeneration: ...}(including mapped options likesize,quality, etc.). - Adds a new black-box request/response test asserting the tool is present in the outgoing request body and that the response still maps back to
DataContent.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tool/hostedtool/hostedtool.go | Adds the ImageGeneration hosted-tool marker and metadata to match the existing hosted tool patterns. |
| provider/openaiprovider/responses.go | Extends the Responses tool-switch to serialize ImageGeneration into params.Tools with supported option mappings. |
| provider/openaiprovider/responses_test.go | Adds a regression test verifying request-side tool serialization for image_generation and output mapping to DataContent. |
|
PratikDhanave (@PratikDhanave) can you resolve the parity gaps? |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
proxy.golang.org
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "proxy.golang.org"See Network Configuration for more information.
Generated by Go API Consistency Review Agent · sonnet46 · 139.9 AIC · ⌖ 4.96 AIC · ⊞ 6K
| AdditionalProperties map[string]any | ||
| } | ||
|
|
||
| func (t *ImageGeneration) Name() string { |
There was a problem hiding this comment.
Cross-repo parity: model field is missing (required in upstream .NET)
The upstream .NET ResponseTool.CreateImageGenerationTool — and its FoundryAITool wrapper — takes model as a required parameter:
public static AITool CreateImageGenerationTool(
string model, // required
ImageGenerationToolQuality? quality = null,
ImageGenerationToolSize? size = null,
ImageGenerationToolOutputFileFormat? outputFileFormat = null,
int? outputCompressionFactor = null,
ImageGenerationToolModerationLevel? moderationLevel = null,
ImageGenerationToolBackground? background = null,
ImageGenerationToolInputFidelity? inputFidelity = null,
ImageGenerationToolInputImageMask? inputImageMask = null,
int? partialImageCount = null)The Go ImageGeneration struct currently has no way to express a model, so callers that rely on a specific image model (e.g. gpt-image-1) cannot specify it through the hosted-tool abstraction. The OpenAI Responses API ToolImageGenerationParam exposes Model as an optional field on the request side, so Go could add a Model string field (populated when non-empty, following the FileSearch.MaximumResultCount precedent) without breaking the untyped map approach for the other settings.
The PR description notes model as a follow-up scope item. Flagging here so it is tracked as a parity gap before the type is considered stable. Happy to defer if the intent is to keep the surface minimal for now, but the gap should be explicit.
Other missing fields (output_compression_factor, moderation, input_fidelity, input_image_mask) are acknowledged in the PR description as explicit follow-ups — no separate comment needed for those.
Cross-Repo Parity ReviewVerdict: ✅ Parity approved — minor documentation gap noted below What was reviewedPR adds Upstream equivalents found
Contract comparisonThe Go Python additionally exposes Example coverage gapPython ships Label status
Adding Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "proxy.golang.org"See Network Configuration for more information.
|
What
Adds
hostedtool.ImageGeneration, a hosted-tool marker mirroring the existingWebSearch,FileSearch,CodeInterpreterandMCPServertypes, and wires it into the OpenAI Responses provider so it is serialized intoparams.Toolsas aresponses.ToolImageGenerationParam.Why
The Responses provider already handles the output side of image generation:
responsesProcessResponseand the streaming path decodeimage_generation_callitems intoDataContent(covered byTestResponsesNonStreamingImageGenerationCall_MapsToDataContent). The underlying SDK also supports the request side viaToolUnionParam.OfImageGeneration. But the tool switch inresponsesBuildCompletionParamshandled onlyFuncTool,WebSearch,FileSearch,CodeInterpreterandMCPServer, so there was no way to actually request image generation through the hostedtool abstraction —params.Toolsnever got animage_generationtool.This closes that input-side gap and brings the Go port in line with the .NET/Python
HostedImageGenerationTool, keeping the hosted-tool surface consistent across SDKs (cross-SDK alignment).How it works
ImageGenerationcarries its optional settings (size,quality,background,output_format,partial_images) via anAdditionalPropertiesmap, exactly likeWebSearch/FileSearch.case *hostedtool.ImageGenerationreads those keys, populates aresponses.ToolImageGenerationParam, and appends aresponses.ToolUnionParam{OfImageGeneration: ...}, following the existingWebSearch/CodeInterpretercases.Tests
TestResponsesImageGenerationTool_MapsToRequestParamsruns through the existing fake-transport harness: it sends&hostedtool.ImageGeneration{AdditionalProperties: {size, quality}}and asserts the outgoing request body contains aimage_generationtool withsize/qualitypopulated (the server-side body assertion is the black-box proxy forparams.Tools), then confirms the response still maps back toDataContent. It fails before the switch case is added (the request omits the tools array) and passes after. Pairs with the existingTestResponsesNonStreamingImageGenerationCall_MapsToDataContent.go build ./...,go vetandgo testpass for the changed packages.Open design questions
map[string]any(size/quality/background/output_format/partial_images) to match theWebSearch/FileSearchprecedent. Is a loosely-typed map the desired long-term shape, or would typed fields onImageGeneration(mirroringFileSearch.MaximumResultCount) be preferred?size,quality,background,output_formatandpartial_images. The SDK also exposesmodel,moderation,input_fidelity,actionandinput_image_mask(inpainting). Happy to add those here or leave as follow-ups.partial_imagestyping: currently acceptsint/int64from the map. If we keep the map surface, should JSON-number semantics (float64) also be accepted?