Skip to content

Conversation

@jverre
Copy link
Collaborator

@jverre jverre commented Nov 21, 2025

Details

Not all models support structured outputs but it's a key strategy we use to ensure the optimizations go smoothly. In this PR, we introduce 2 new strategies to support all models for optimization:

  1. Native support
  2. Tool calling support
  3. Prompting support

Change checklist

  • User facing
  • Documentation update

Issues

  • Resolves #
  • OPIK-000

Testing

Added tests

Documentation

N/A

Copilot AI review requested due to automatic review settings November 21, 2025 11:38
@jverre jverre requested review from a team, dsblank and vincentkoc as code owners November 21, 2025 11:38
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces improved structured output support for the optimizer by implementing three strategies to handle models with varying capabilities: native response format support, tool calling support, and prompt injection. This ensures the optimizer can work with all models, even those without native JSON schema support.

Key Changes

  • Added three structured output strategies (native, tool_call, prompt_injection) to accommodate different model capabilities
  • Refactored parameter handling to extract and validate response_format_type from model parameters
  • Enhanced error messages to guide users toward alternative strategies when parsing fails

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
sdks/opik_optimizer/src/opik_optimizer/_llm_calls.py Core implementation of three structured output strategies with helper functions for tool definitions, prompt injection, and response extraction
sdks/opik_optimizer/tests/unit/test_llm_calls.py Unit tests for all three strategies using mocked litellm responses
sdks/opik_optimizer/tests/e2e/test_llm_calls_integration.py End-to-end integration tests verifying each strategy with real model calls and trace validation
sdks/opik_optimizer/tests/conftest.py Test fixtures recreated from Python SDK for fake backend and trace capture
sdks/opik_optimizer/pyproject.toml Added mypy ignore configuration for testlib module

vincentkoc
vincentkoc previously approved these changes Nov 21, 2025
dsblank
dsblank previously approved these changes Nov 21, 2025
Copy link
Contributor

@dsblank dsblank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jverre jverre dismissed stale reviews from dsblank and vincentkoc via 98514a4 December 4, 2025 14:18
@github-actions github-actions bot added dependencies Pull requests that update a dependency file python Pull requests that update Python code Optimizer SDK tests Including test files, or tests related like configuration. labels Dec 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file Optimizer SDK python Pull requests that update Python code tests Including test files, or tests related like configuration.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants