Skip to content

Add Groq agent tool cassette regressions#2011

Merged
gold-silver-copper merged 1 commit into
mainfrom
test/groq-cassette-regression
Jul 4, 2026
Merged

Add Groq agent tool cassette regressions#2011
gold-silver-copper merged 1 commit into
mainfrom
test/groq-cassette-regression

Conversation

@gold-silver-copper

@gold-silver-copper gold-silver-copper commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a broad cassette-backed Groq regression suite for Rig's OpenAI-compatible Groq chat-completions path.

New scenarios under tests/providers/groq/agent_tool_sessions.rs cover:

  • long multi-turn sequential tool loops with five local tools;
  • streaming/non-streaming parity for equivalent multi-tool sessions;
  • multiple tool calls in a single assistant turn with parallel_tool_calls;
  • raw streaming tool-call deltas and final object-valued tool calls;
  • long caller-owned history replay with system/user/assistant/tool-result messages;
  • ToolChoice::{Auto, Required, Specific, None} wire behavior;
  • JSON object response-format roundtrip;
  • native JSON-schema structured output roundtrip;
  • low-latency text streaming termination and final usage accounting.

Models used

  • llama-3.3-70b-versatile: primary stable Groq-hosted chat/tool model for tool loops, streaming, JSON object mode, usage, and metadata.
  • openai/gpt-oss-20b: JSON-schema structured-output scenario because Groq rejects json_schema response format for llama-3.3-70b-versatile.

Inspiration references

Reviewed these implementation/test patterns in many_rigs/inspirations:

  • vercel-ai-sdk/packages/groq/src/groq-chat-language-model.ts
  • vercel-ai-sdk/packages/groq/src/groq-chat-language-model.test.ts
  • vercel-ai-sdk/packages/groq/src/groq-prepare-tools.ts
  • vercel-ai-sdk/packages/groq/src/groq-prepare-tools.test.ts
  • pydantic-ai/pydantic_ai_slim/pydantic_ai/models/groq.py
  • pydantic-ai/tests/models/test_groq.py
  • pydantic-ai/tests/providers/test_groq.py
  • langchain/libs/partners/groq/langchain_groq/chat_models.py
  • langchain/libs/partners/groq/tests/integration_tests/test_chat_models.py

Those references motivated coverage for specific tool choice, Groq response_format, streaming x_groq.usage, reasoning-token usage details, and Groq timing metadata.

Bugs found and fixed

  • Groq max_tokens was not forwarded in chat-completions requests.
  • Groq ToolChoice::Specific incorrectly reused the generic OpenAI Chat Completions conversion, which rejected specific tool choices even though Groq supports a single function-choice object.
  • Rig warned and ignored output_schema for Groq; Groq now maps Rig output schemas to response_format: { type: "json_schema", ... }.
  • Groq streaming usage can arrive via x_groq.usage; Rig now reads that path in addition to terminal usage chunks.
  • Groq/OpenAI-compatible raw usage now preserves completion_tokens, completion_tokens_details.reasoning_tokens, and Groq timing fields (queue_time, prompt_time, completion_time, total_time) and maps output token usage from completion_tokens when present.
  • Boxed Copilot's chat raw-response enum variant to keep CI clippy green under -D warnings after the latest toolchain started flagging large_enum_variant.

Validation

Record-mode commands run with GROQ_API_KEY from the environment:

RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test groq sequential_complex_tool_calls_nonstreaming -- --nocapture --test-threads=1
RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test groq agent_tool_sessions -- --nocapture --test-threads=1
RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test groq sequential_complex_tool_calls_streaming -- --nocapture --test-threads=1
RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test groq tool_choice_auto_required_specific_and_none -- --nocapture --test-threads=1
RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test groq long_history_replay_with_tool_result_continuation -- --nocapture --test-threads=1
RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test groq json_schema_structured_output_roundtrip -- --nocapture --test-threads=1

Replay/safety/unit checks:

env -u GROQ_API_KEY cargo test -p rig --all-features --test groq agent_tool_sessions -- --nocapture --test-threads=1
cargo test -p rig --all-features --test groq cassette_safety -- --nocapture --test-threads=1
env -u GROQ_API_KEY cargo test -p rig --all-features --test groq groq -- --nocapture --test-threads=1
cargo test -p rig-core providers::groq -- --nocapture
cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings

cargo test was also attempted and reached the pre-existing unrelated OpenAI cassette mismatch in openai::cassette::permission_control::permission_control_streaming_example.

Manual cassette inspection

  • Confirmed Groq cassette requests use the expected /openai/v1/chat/completions path and expected request bodies.
  • Confirmed no API keys, bearer tokens, cookies, account identifiers, or rate-limit error payloads are present in tests/cassettes/groq/.
  • Confirmed cassettes preserve Groq response metadata such as system_fingerprint, usage, x_groq, and timing fields where the live API surfaced them.
  • Confirmed replay passes without GROQ_API_KEY.

Non-goals / known gaps

  • Does not migrate the older ignored Groq smoke tests to cassettes.
  • Does not cover Groq transcription or native browser-search/tooling.
  • Does not record Groq model-rejection behavior beyond using openai/gpt-oss-20b for JSON-schema output after observing llama-3.3-70b-versatile reject that response format.

@gold-silver-copper gold-silver-copper force-pushed the test/groq-cassette-regression branch from ef29a14 to 13a5d08 Compare July 4, 2026 18:31
@gold-silver-copper gold-silver-copper added this pull request to the merge queue Jul 4, 2026
Merged via the queue into main with commit ea014f3 Jul 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant