Multi round debat#35
Merged
Merged
Conversation
…m-model vs threat-model terminology
- Add RedBlueDebateEngine: multi-round Red (attack) vs Blue (defence) LLM
debate over the top-N GDAF attack scenarios, adjusting path_score/risk_level
in place and surfacing a new HTML report section. Opt-in via
config/ai_config.yaml (debate.enabled: false by default), degrades
silently when AI is offline or no GDAF scenarios exist.
- Gate the Graphical Editor behind a new --graphical-editor CLI flag,
disabled by default. The /graphical route and its API now 404 and the
menu link is hidden unless explicitly enabled.
- Rename the DSL title convention from "# Threat Model: {Name}" to
"# System Model: {Name}" across docs, prompts, and generated content —
the DSL describes the system; SecOpsTM generates the threat model from
it. The legacy tag is still parsed with a deprecation warning
(get_model_name). Migrate all 30 threatModel_Template/*.md files to the
new tag.
- Fix two broken "Maintenance" badge URLs and the stale version string in
README.md / CLAUDE.md.
test blue / red debate
Root cause across most of these: several report sections and export formats
were computed from a stale or wrong data source instead of the final
AI-enriched, debate-aware threat list, or read config keys that were never
actually wired to any code.
AI enrichment pipeline
- ReportGenerator now delegates to AIService._enrich_with_ai_threats()
instead of its own thinner duplicate. This restores the per-component
AIThreatCache (previously never invoked, so every run re-queried the LLM)
and the SOC analysis pass (previously always empty in the HTML report).
- threat_generation.confidence_threshold and max_threats_per_component,
previously unread, now filter/cap AI-generated threats per component.
- rag.top_k_examples, previously unread, now controls RAG retrieval k.
- CLI --project mode now builds and passes an AIService to
generate_project_reports(), so cross-model RAG threat generation runs in
CLI mode too (previously web-UI only).
Post-debate consistency (GDAF)
- .afb Attack Flow files are re-written after the Red/Blue debate mutates
scenario scores, instead of staying frozen at pre-debate values.
- Threat IDs (T-0001 style) are assigned before CISO triage/threat-graph/
debate run, so all of them reference the same IDs as the final report
instead of the LLM inventing its own scheme.
- CISO triage and the interactive threat graph now run after the debate and
can reference GDAF scenarios and debate outcomes (new "Automatically
Discovered Attack Paths" section mirrors this for the GDAF-independent
AttackFlowGenerator).
- JSON/STIX/Navigator/AttackFlowGenerator exports reuse the AI-enriched
threat list cached by generate_html_report() via a new get_enriched_threats()
helper, instead of recomputing a pytm-only list.
Other correctness fixes
- Severity multipliers (## Severity Multipliers DSL section) are now synced
from the parsed model instead of read from the bundled example template —
previously affected CLI single-file runs and the entire server/web UI.
- MitreMapping no longer dumps the full ~150-entry STRIDE-category CAPEC
bucket onto every threat lacking explicit capec_ids; ranks by description
keyword overlap and caps at 10.
- Diagram severity heat map (export_service.py) is now computed after AI
enrichment instead of before, so it can reflect AI-derived severities.
Cleanup
- Removed _expand_class_targets(), dead since process_threats() switched to
tm.findings; corrected architecture.md/decisions.md which still documented
it as active.
- Removed dead ai_config.yaml keys (attack_flows.format/max_depth/
include_detection_points, threat_generation.include_attack_flows) that had
no implementation; wired attack_flows.enabled/generate_for_categories into
AttackFlowGenerator instead.
Regression tests added throughout for every behavior change.
…ion, concurrency)
- ai_service.py: skip system-level RAG below rag.min_components (default 3) —
tiny models mostly got restated single-component threats instead of genuine
cross-boundary chaining; applies to both live enrichment and project-mode
cross-model RAG.
- attack_id_validator.py: extend AttackIdValidator to also validate
capecs[].capec_id and defend_mitigations[].id against the committed CAPEC
catalog CSV and d3fend.csv, not just ATT&CK technique IDs — LLM-cited CAPEC
IDs were previously only format-checked ("CAPEC-" prefix), never verified
against real data before appearing in a report with a stable T-000N ID or
in STIX export. CWE intentionally excluded: the pipeline never reads the
LLM's raw cwe_ids field, so there's nothing to ground-truth.
- diagram_generator.py: replace hardcoded /tmp/graphviz_error.log with
tempfile.mkstemp() — avoids concurrent requests clobbering each other's
log and symlink-attack risk on a shared host.
- diagram_generator.py / svg_generator.py: replace bare `except:` (which
swallows KeyboardInterrupt) with `except Exception` + logging.
- server.py: move initial_model_file_path/initial_project_path from
process-wide globals to Flask session state (threaded=True means two
concurrent tabs/users could otherwise redirect each other's exports to the
wrong model); log and continue instead of silently dropping SSE queue
push failures; warn at startup when binding to a non-loopback host, since
no route has authentication.
- README.md / DOCKER_HUB.md: default `docker run` examples to
`-p 127.0.0.1:5000:5000` with a short security note, since the Docker
image sets FLASK_HOST=0.0.0.0 by default.
- data_loader.py: add load_capec_catalog_ids(), parsing the full CAPEC ID
set from the committed CSV.
- Tests: cover all of the above (RAG threshold, CAPEC/D3FEND validation,
tempfile log path, except-clause logging, session isolation, startup
warning) plus new coverage for previously-untested core/model_factory.py,
core/stride_constants.py, and generation/tactic_logic.py.
Full suite: 1417 passed.
Adds a short, grounded narrative and business-impact summary to each automatically discovered attack path in the HTML report. Context: AttackFlowGenerator.get_paths_summary() finds the best (highest-severity) attack path per STRIDE category by following each threat's MITRE ATT&CK tactic progression through the architecture — pure graph traversal over the fully consolidated threat list (pytm + AI + LLM sources), no GDAF context required, capped at one path per STRIDE category (≤6 total). What's new: each of those paths now gets a short narrative explaining how an attacker would realistically move through it, plus a business-impact paragraph — generated by a dedicated LLM persona (attack_path_narrative in config/prompts.yaml), rendered as an expandable row under the path in the "Automatically Discovered Attack Paths" report section. Hallucination guard (two layers): 1. Minimal attack surface — the prompt only gives the model technique names (e.g. "Exploit Public-Facing Application"), never IDs. Nothing to copy incorrectly. 2. Code-level rejection, not just a prompt instruction — _narrative_has_id_leakage() regex-scans the response for anything shaped like an ATT&CK (T####), CVE, CAPEC, or D3FEND ID. A single match discards the entire response — fail closed, no partial trust, since one fabricated ID means the rest of the prose can't be trusted either. Config: attack_flows.include_narrative in ai_config.yaml (default true) — skips silently when AI is offline or disabled. Docs updated: README (AI Roles table), decisions.md (design rationale — reject-on-any-match vs. corpus validation, why no top_n gate needed), architecture.md (new pipeline section 5d), docs/features.md. Tests: 24 new tests in tests/test_attack_path_narrative.py (regex unit tests, gating, grounding-violation rejection on both narrative and business_impact, multi-path handling, non-fatal provider errors) + 1 in test_attack_flow_generator.py. Full suite: 1438 passed.
update report template
add groq and xai provider
fix pytm version
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
debate over the top-N GDAF attack scenarios, adjusting path_score/risk_level
in place and surfacing a new HTML report section. Opt-in via
config/ai_config.yaml (debate.enabled: false by default), degrades
silently when AI is offline or no GDAF scenarios exist.
disabled by default. The /graphical route and its API now 404 and the
menu link is hidden unless explicitly enabled.
"# System Model: {Name}" across docs, prompts, and generated content —
the DSL describes the system; SecOpsTM generates the threat model from
it. The legacy tag is still parsed with a deprecation warning
(get_model_name). Migrate all 30 threatModel_Template/*.md files to the
new tag.
README.md / CLAUDE.md.