Keep eligibility assessment enabled when CoT is disabled

[majdabd]

Disabling use_cot_reasoning previously skipped the entire eligibility-assessment stage, despite both model processors supporting direct JSON assessment. The flowchart also presented assessment as an optional side branch.

This change makes rag.enabled the shared stage gate for runtime and preflight. use_cot_reasoning selects the CoT or direct JSON prompt, with assessment still enabled by default. The SVGs now show assessment on the main workflow and an explicit retrieval-only bypass.

Ranked results record assessment controls, mode, output availability, and assessed trial IDs. Reports label retrieval-only/partial results and suppress old assessments and reasoning text from prior runs. Switching assessment controls invalidates match completion and forces regeneration of per-trial outputs; unchanged, successful results retain resume. The shortlist budget is included in the controls. Partial or unavailable assessments stay pending, and retries reuse only compatible successful outputs. Deleted/corrupt assessment outputs also trigger a retry. Each attempt stages its outputs separately so early returns and failed writes cannot revive prior verdicts. Reranking preserves the metadata and cannot revive old assessments in a retrieval-only result. An all-missing/error assessment result uses the retrieval fallback.

Migration

Set rag.enabled: false to disable assessment. use_cot_reasoning: false alone now selects direct JSON assessment. Legacy matches without the new Run metadata are recomputed when matching resumes. The README and changelog mark this behavior as unreleased after 0.9.0; this PR does not publish another package version.

Validation

• Full local suite: 498 passed.
• Isolated installed-wheel CLI smoke passed, including explicit retrieval-only metadata, HTML labels, report repair, and unchanged ranking on resume.
• Tests cover both backend dispatch paths with deterministic processors, preflight flag combinations, mode/prompt-style switches, stale-output suppression, missing/partial outputs, metadata-preserving reranking, legacy migration, shortlist-budget changes, incomplete-assessment retries, missing saved outputs, and aborted/failed-write isolation.
• Ruff, staged gitleaks, strict MkDocs, wheel/sdist build, SVG rendering, and a headless Firefox inspection of the synthetic report passed.

This binds assessment controls only. Complete patient/corpus/model evidence identity, clinical completeness/decision semantics, and actual GPU qualification remain roadmap work. No benchmark improvement is claimed.

[Copilot]

🟡 Changes recommended

One critical and two moderate unresolved review findings must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR decouples eligibility assessment from CoT prompting, adds direct-JSON assessment, provenance-aware resume behavior, retrieval-only reporting, and related documentation.

Changes:

• Uses rag.enabled as the assessment gate.
• Records assessment controls, modes, outputs, and trial IDs.
• Adds stale-output suppression, fallback handling, reranking metadata preservation, and expanded validation.

File summaries

File	Summary
tests/test_resume_and_atomicity.py	Updates resume and completion-marker coverage.
tests/test_audit_fixes_batch8.py	Updates resume fixtures.
tests/test_assessment_modes.py	Adds assessment-mode and resume coverage.
src/trialmatchai/services/preflight.py	Shares assessment gating.
src/trialmatchai/orchestration.py	Tracks controls for resume invalidation.
src/trialmatchai/matching/trial_ranker.py	Preserves run metadata during ranking.
src/trialmatchai/matching/eligibility_base.py	Supports forced output regeneration.
src/trialmatchai/matching/assessment.py	Defines assessment settings and provenance; retains two moderate issues concerning shortlist-budget invalidation and unavailable-run retryability.
src/trialmatchai/main.py	Dispatches assessment modes and fallback ranking; has a critical stale-output path after early assessment returns.
src/trialmatchai/interop/exporters/templates/report.html	Displays assessment status labels.
src/trialmatchai/interop/exporters/html_report.py	Suppresses stale or unavailable assessment data.
src/trialmatchai/config/settings.py	Documents independent assessment controls.
src/trialmatchai/config/config.json	Enables assessment explicitly by default.
scripts/installed_smoke.py	Validates retrieval-only metadata and reports.
README.md	Documents modes and migration.
docs/production-validation.md	Records validation results.
docs/pipeline.md	Documents assessment controls and provenance.
docs/index.md	Updates the workflow overview.
docs/assets/readme-overview.svg	Shows default assessment behavior.
docs/assets/matching-flow.svg	Shows assessment and retrieval-only paths.
CHANGELOG.md	Records unreleased behavior changes.

Review details

Suppressed comments (1)

src/trialmatchai/matching/assessment.py:50

• When assessment remains enabled but every per-trial response is missing or an error, the new run is recorded as assessment_status: "unavailable" with a retrieval fallback, but this predicate still marks it complete as soon as the controls match. A subsequent --resume therefore skips the patient and never retries the error sidecars, even though BaseTrialProcessor.process_trials explicitly treats those files as retryable. Keep enabled/unavailable runs pending (while retaining no_candidates as a terminal state) so assessment can be regenerated.

        return (
            isinstance(result.get("RankedTrials"), list)
            and run.get("schema_version") == 1
            and run.get("assessment") == assessment_settings(config)
        )

• Files reviewed: 19/21 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.