-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[NA] [Optimizer] Improved structured output support #4169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces improved structured output support for the optimizer by implementing three strategies to handle models with varying capabilities: native response format support, tool calling support, and prompt injection. This ensures the optimizer can work with all models, even those without native JSON schema support.
Key Changes
- Added three structured output strategies (
native,tool_call,prompt_injection) to accommodate different model capabilities - Refactored parameter handling to extract and validate
response_format_typefrom model parameters - Enhanced error messages to guide users toward alternative strategies when parsing fails
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
sdks/opik_optimizer/src/opik_optimizer/_llm_calls.py |
Core implementation of three structured output strategies with helper functions for tool definitions, prompt injection, and response extraction |
sdks/opik_optimizer/tests/unit/test_llm_calls.py |
Unit tests for all three strategies using mocked litellm responses |
sdks/opik_optimizer/tests/e2e/test_llm_calls_integration.py |
End-to-end integration tests verifying each strategy with real model calls and trace validation |
sdks/opik_optimizer/tests/conftest.py |
Test fixtures recreated from Python SDK for fake backend and trace capture |
sdks/opik_optimizer/pyproject.toml |
Added mypy ignore configuration for testlib module |
dsblank
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Details
Not all models support structured outputs but it's a key strategy we use to ensure the optimizations go smoothly. In this PR, we introduce 2 new strategies to support all models for optimization:
Change checklist
Issues
Testing
Added tests
Documentation
N/A