feat: disable reasoning mode for AI threat generation#33
Merged
ellipse2v merged 1 commit intoJun 14, 2026
Conversation
Some modern LLMs spend a significant portion of max_tokens on internal reasoning, leaving insufficient budget for structured JSON output. This causes batch threat generation responses to truncate, producing invalid JSON that cannot be parsed. Pass enable_thinking=False to LiteLLM acompletion() calls so that 100% of the token budget goes to content generation.
Owner
|
Thanks for the fix — the root cause analysis is correct, reasoning tokens competing with the output budget is exactly the kind of issue that causes silent JSON truncation in batch generation. I'll merge this as-is. As a follow-up I'll make |
ellipse2v
approved these changes
Jun 14, 2026
ellipse2v
added a commit
that referenced
this pull request
Jun 14, 2026
enable_thinking was hardcoded to False in litellm_client.py (PR #33). This moves it to a per-provider config field so Anthropic extended thinking can be enabled when a sufficient max_tokens budget is available. threat_analysis/ai_engine/providers/litellm_client.py - generate_content(): replace hardcoded False with bool(self.provider_config.get("enable_thinking", False)) — reads from ai_config.yaml, defaults to False - check_connection(): remove enable_thinking=False — the health ping uses max_tokens=1 and never benefits from reasoning mode config/ai_config.yaml - Add enable_thinking as a commented-out field under the mistral provider block, with a note that it should only be set to true alongside a large max_tokens budget tests/test_litellm_provider.py - test_enable_thinking_defaults_to_false: asserts acompletion receives enable_thinking=False when the field is absent from provider config - test_enable_thinking_true_from_provider_config: asserts enable_thinking=True is forwarded to acompletion when explicitly set in provider config - test_check_connection_does_not_pass_enable_thinking: asserts enable_thinking is absent from the health check call regardless of provider config
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reasoning tokens consume the same max_tokens budget as content tokens, not a separate quota. When reasoning is enabled, most of the budget is allocated to internal reasoning, leaving severely limited headroom for actual output. Under this constraint, batch threat generation responses are truncated mid-JSON and cannot be parsed.
Set enable_thinking=False so the full token budget is available for content generation, resolving the truncation.