Add WebP support for Google GenAI and Vertex AI#6526
Open
jialiuyang wants to merge 1 commit into
Open
Conversation
Gemini chat models and Vertex AI Multimodal Embedding both accept image/webp inputs, but Spring AI rejected them before they could reach the provider: * MimeTypeDetector registered only png, jpeg, jpg and gif, so any .webp resource passed to Gemini chat clients raised "Unable to detect the MIME type" during URL, URI, file or resource detection. * VertexAiMultimodalEmbeddingModel rejected image/webp media at the SUPPORTED_IMAGE_MIME_SUB_TYPES check, throwing "Unsupported image mime type" before any embedding request was issued. This adds image/webp to both code paths, updates the MimeTypeDetector class javadoc, and extends the parameterized MimeTypeDetectorTests so the new mapping is exercised through every detection entry point. Fixes spring-projects#6474 Signed-off-by: jialiuyang <495120021@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gemini chat models and Vertex AI Multimodal Embedding both accept
image/webpinputs, but Spring AI rejected those payloads before they could reach the provider.Problem
MimeTypeDetector(spring-ai-google-genai) registered onlypng,jpeg,jpgandgif, so any.webpURL/URI/File/Path/Resource/Stringpassed to a Gemini chat client raisedUnable to detect the MIME type. Its class javadoc also listed onlyimage/gif,image/pngandimage/jpegas supported, which contradicts the upstream Gemini model reference, which has acceptedimage/webpsince 1.5.VertexAiMultimodalEmbeddingModelallowedimage/jpeg,image/gif,image/pngandimage/bmp, and rejectedimage/webpatSUPPORTED_IMAGE_MIME_SUB_TYPES, throwingUnsupported image mime typeeven though the Vertex multimodal embedding API supports WebP.The use case in the linked issue - pre-processing media into WebP and then sending it to a multimodal model through Spring AI - therefore fails on the Spring AI side, not at the provider.
Changes
MimeTypeDetector: registerwebp?image/webpand addimage/webpto the supported-types list in the class javadoc.MimeTypeDetectorTests: includeWEBPin the uppercase-extension parameter set andtest.webpin the cross-method consistency parameter set. The existing parameterized tests sourced fromGEMINI_MIME_TYPES(URL, URI,File,Path,Resource,String) automatically cover the new mapping.VertexAiMultimodalEmbeddingModel: addimage/webptoSUPPORTED_IMAGE_MIME_SUB_TYPES.No public API was added or changed; existing supported MIME types still behave exactly as before.
Testing
./mvnw -pl models/spring-ai-google-genai -am test- parameterizedMimeTypeDetectorTestsexercise URL/URI/File/Path/Resource/String detection for the newwebpmapping. Integration tests against live Gemini/Vertex were not run.Fixes #6474