Skip to content

feat(openai,dmr): switch thinking off on OpenAI-compatible endpoints and add provider_opts.extra_body - #4310

Open
IsmaelMartinez wants to merge 5 commits into
docker:mainfrom
IsmaelMartinez:feat/thinking-off-openai-compatible
Open

IsmaelMartinez wants to merge 5 commits into
docker:mainfrom
IsmaelMartinez:feat/thinking-off-openai-compatible

Conversation

@IsmaelMartinez

Copy link
Copy Markdown
Contributor

Open-weight reasoning models served through OpenAI-compatible endpoints think by default and each server exposes a request field to turn it off, but the openai client only honoured thinking_budget for OpenAI reasoning model names, applyModelDefaults dropped a disabled budget before any client saw it, and no provider_opts beyond the sampling allowlist reached the wire, so a Qwen3 model on mlx_lm, llama.cpp, vLLM or SGLang reasoned on every call and could spend the whole max_tokens budget before answering. This adds provider_opts.extra_body, an object merged verbatim into the chat-completions body for the openai and dmr clients on any provider (an explicit key wins over anything derived), and makes thinking_budget: none/0 on a user-supplied base_url send chat_template_kwargs.enable_thinking=false on Chat Completions with the existing 256-token output floor; the dmr client sends the same per request alongside llamacpp.reasoning-budget, which now actually reaches _configure for none. The switch is strictly opt-in: it needs the user's own none/0 (resolved once in the factory as options.WithChatTemplateThinkingOff, so title-generation and compaction clones inherit it but cannot trigger it alone), compares the resolved base_url against the alias default so built-in aliases and their clones are unaffected, and is never sent for azure, chatgpt or OpenAI model names behind a proxy. Docs, agent-schema.json and examples/thinking_budget.yaml are updated, including a note that reasoning tokens count against max_tokens.

Refs #4309

@aheritier aheritier added area/config For configuration parsing, YAML, environment variables area/docs Documentation changes area/models LLM model integrations and model providers area/providers/openai For features/issues/fixes related to the usage of OpenAI models area/providers/docker-model-runner Docker Model Runner (DMR) local inference kind/feat PR adds a new feature (maps to feat:). Use on PRs only. labels Sep 16, 2026
@IsmaelMartinez
IsmaelMartinez marked this pull request as ready for review September 16, 2026 11:42
@IsmaelMartinez
IsmaelMartinez requested a review from a team as a code owner September 16, 2026 11:42
@lyonsno

lyonsno commented Sep 17, 2026

Copy link
Copy Markdown

I reproduced a DMR side effect of preserving the disabled budget: MCP sampling clones now send llamacpp.reasoning-budget: 0 to the model-wide _configure endpoint.

Using the real registry and CloneWithOptions with the options from samplingModelOptionsFor, an agent configured with budget 4096 produces:

base 032d705: initial configure 4096; sampling configure omits budget
head 6e0a73b: initial configure 4096; sampling configure sets budget 0

Title/compaction clones skip that second configure call; sampling clones don't. Model Runner stores accepted configuration by model, so on the llama.cpp route this can leave the main agent using a zero reasoning budget after sampling. The original provider's config still says 4096, and its completion path doesn't reconfigure the runner.

Could sampling clones also avoid model-wide reconfiguration and use the new per-request thinking-off field? I verified the HTTP writes on both revisions; the downstream effect is traced through Model Runner's scheduler/loader, not a live-model run. The parent already reconfigures with the budget omitted—the newly introduced part is explicitly setting it to zero.

@aheritier aheritier added the status/needs-rebase PR has merge conflicts or is out of date with main label Sep 17, 2026
@aheritier

Copy link
Copy Markdown
Collaborator

👋 This PR has merge conflicts with the base branch. Please rebase or merge the latest base branch and resolve them. I've moved it to draft and added status/needs-rebase; it'll be picked back up automatically once the conflicts are cleared.

@IsmaelMartinez
IsmaelMartinez force-pushed the feat/thinking-off-openai-compatible branch from 6e0a73b to 80e9385 Compare September 18, 2026 08:38
@aheritier aheritier added status/needs-rebase PR has merge conflicts or is out of date with main and removed status/needs-rebase PR has merge conflicts or is out of date with main labels Sep 18, 2026
…and add provider_opts.extra_body

thinking_budget: none (or 0) on a user-supplied base_url now sends chat_template_kwargs.enable_thinking=false for non-OpenAI model names, and the dmr client sends it per request instead of only for NoThinking clones; provider_opts.extra_body merges arbitrary JSON fields into the chat completion body on any provider for vendors with their own off switch.
…owards OpenAI-hosted endpoints

The chat_template_kwargs switch now requires the user's own thinking_budget none/0 (threaded as options.WithThinkingDisabled so title-generation and compaction clones inherit it but cannot trigger it alone), skips azure/chatgpt and OpenAI model names via modelinfo.IsOpenAIHosted, tolerates a trailing slash when comparing an alias base_url, and is preserved for Chat Completions only.
…bit and trim comments

Merge the custom base_url and disabled-thinking options into a single ChatTemplateThinkingOff bit computed by one predicate shared with keepsDisabledThinking, flatten the OpenAI client switch, allocate extras lazily and skip the sampling loop when provider_opts is empty, inline the env expander, dedupe the openai and dmr test drivers, and cut every added comment to one sentence.

Refs docker#4309
@IsmaelMartinez
IsmaelMartinez force-pushed the feat/thinking-off-openai-compatible branch from 80e9385 to 9d7fcb2 Compare September 19, 2026 07:22
@aheritier aheritier removed the status/needs-rebase PR has merge conflicts or is out of date with main label Sep 19, 2026
Model Runner stores the accepted configuration per model, so a no-thinking clone sending its own reasoning budget on _configure sticks to the main agent; skip it like the title and compaction clones do and rely on the per-request switch.
@IsmaelMartinez

Copy link
Copy Markdown
Contributor Author

Applied in a38a765dc. Model Runner replaces a model's stored configuration on every _configure, so the parent's budget-omitted reconfigure already wiped the agent's 4096 on main (docker model configure showConfig: {}, verified live on llama.cpp b9879); this PR changed the wiped value to 0. dmr.NewClient now skips _configure for NoThinking clones, matching title and compaction clones, and the sampling request keeps its per-request chat_template_kwargs. WithNoThinking is set only by the sampling and title clones, so nothing else changes. TestNewClientClonesSkipConfigure covers the main agent (configures 4096) and the three clones (no configure); through the real registry the scenario now produces one configure call instead of two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config For configuration parsing, YAML, environment variables area/docs Documentation changes area/models LLM model integrations and model providers area/providers/docker-model-runner Docker Model Runner (DMR) local inference area/providers/openai For features/issues/fixes related to the usage of OpenAI models kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants