feat(openai,dmr): switch thinking off on OpenAI-compatible endpoints and add provider_opts.extra_body - #4310
IsmaelMartinez wants to merge 5 commits into
Conversation
|
I reproduced a DMR side effect of preserving the disabled budget: MCP sampling clones now send Using the real registry and Title/compaction clones skip that second configure call; sampling clones don't. Model Runner stores accepted configuration by model, so on the llama.cpp route this can leave the main agent using a zero reasoning budget after sampling. The original provider's config still says 4096, and its completion path doesn't reconfigure the runner. Could sampling clones also avoid model-wide reconfiguration and use the new per-request thinking-off field? I verified the HTTP writes on both revisions; the downstream effect is traced through Model Runner's scheduler/loader, not a live-model run. The parent already reconfigures with the budget omitted—the newly introduced part is explicitly setting it to zero. |
|
👋 This PR has merge conflicts with the base branch. Please rebase or merge the latest base branch and resolve them. I've moved it to draft and added |
6e0a73b to
80e9385
Compare
…and add provider_opts.extra_body thinking_budget: none (or 0) on a user-supplied base_url now sends chat_template_kwargs.enable_thinking=false for non-OpenAI model names, and the dmr client sends it per request instead of only for NoThinking clones; provider_opts.extra_body merges arbitrary JSON fields into the chat completion body on any provider for vendors with their own off switch.
…owards OpenAI-hosted endpoints The chat_template_kwargs switch now requires the user's own thinking_budget none/0 (threaded as options.WithThinkingDisabled so title-generation and compaction clones inherit it but cannot trigger it alone), skips azure/chatgpt and OpenAI model names via modelinfo.IsOpenAIHosted, tolerates a trailing slash when comparing an alias base_url, and is preserved for Chat Completions only.
…bit and trim comments Merge the custom base_url and disabled-thinking options into a single ChatTemplateThinkingOff bit computed by one predicate shared with keepsDisabledThinking, flatten the OpenAI client switch, allocate extras lazily and skip the sampling loop when provider_opts is empty, inline the env expander, dedupe the openai and dmr test drivers, and cut every added comment to one sentence. Refs docker#4309
80e9385 to
9d7fcb2
Compare
Model Runner stores the accepted configuration per model, so a no-thinking clone sending its own reasoning budget on _configure sticks to the main agent; skip it like the title and compaction clones do and rely on the per-request switch.
|
Applied in |
Open-weight reasoning models served through OpenAI-compatible endpoints think by default and each server exposes a request field to turn it off, but the openai client only honoured
thinking_budgetfor OpenAI reasoning model names,applyModelDefaultsdropped a disabled budget before any client saw it, and noprovider_optsbeyond the sampling allowlist reached the wire, so a Qwen3 model on mlx_lm, llama.cpp, vLLM or SGLang reasoned on every call and could spend the wholemax_tokensbudget before answering. This addsprovider_opts.extra_body, an object merged verbatim into the chat-completions body for the openai and dmr clients on any provider (an explicit key wins over anything derived), and makesthinking_budget: none/0on a user-suppliedbase_urlsendchat_template_kwargs.enable_thinking=falseon Chat Completions with the existing 256-token output floor; the dmr client sends the same per request alongsidellamacpp.reasoning-budget, which now actually reaches_configurefornone. The switch is strictly opt-in: it needs the user's ownnone/0(resolved once in the factory asoptions.WithChatTemplateThinkingOff, so title-generation and compaction clones inherit it but cannot trigger it alone), compares the resolvedbase_urlagainst the alias default so built-in aliases and their clones are unaffected, and is never sent forazure,chatgptor OpenAI model names behind a proxy. Docs,agent-schema.jsonandexamples/thinking_budget.yamlare updated, including a note that reasoning tokens count againstmax_tokens.Refs #4309