eval: keep the answer text, and count pages named in prose but not cited

[swati510]

Auditing the confident-but-wrong answers from the last run turned up two gaps in what a result blob records.

The answer text was thrown away. The blob kept path lists only, so deciding whether a confident answer was genuinely wrong — or the expectation too narrow — meant re-asking every one of them against the index and hoping the model repeated itself. It is the cheapest field in the blob and the only one a reviewer can argue with. Now kept whole, per question, empty string on an abstention.

One failure was wearing another's clothes. An audited answer named the expected file in its prose, with the right symbol, and cited two different files. That scores zero — identically to an answer that never found the page. But it is a citation failure, not a retrieval one, and the two need opposite fixes. Each question now records the expected paths its answer named without putting them in front of the caller (expected_paths_named_only_in_prose), the run counts them (n_answers_naming_an_uncited_expected_path, also in the CLI summary line), and each one logs a warning as it happens.

The prose check is a plain substring match on the file part of the path, and it is deliberately loose — it says "go and read this one", it is never a score. A path already in the citations or the retrieval block can never appear in it.

Tests. 6 new, failing before and passing after: the answer reaching the blob, an abstention kept as empty text rather than dropped, a prose-only mention recorded and warned about, a cited path never reported as prose-only, a ::symbol expectation counted when its file is named, and a plain miss naming nothing.

python -m pytest tests/test_answer_eval_*.py → 137 passed (was 131).

[repowise-bot]

✅ Health: 9.9 (unchanged)

📋 At a glance
3 hotspots touched · 1 new finding introduced.

Files & modules (2)

• answer_eval (2 files)
  • answer_eval/__main__.py
  • answer_eval/runner.py
• tests (1 file)
  • tests/test_answer_eval_runner.py

🔎 More signals (1)

🔥 Hotspots touched (3)

• answer_eval/__main__.py — 1 commits/90d, 0 dependents · primary owner: Swati Ahuja (100%)
• answer_eval/runner.py — 1 commits/90d, 2 dependents · primary owner: Swati Ahuja (100%)
• tests/test_answer_eval_runner.py — 1 commits/90d, 0 dependents · primary owner: Swati Ahuja (100%)

---

📊 Full report · ⭐ Star Repowise · 📥 Install bot · Last updated 2026-07-29 17:33 UTC
_{Silence on a single PR with [skip repowise] in the title · Per-repo toggle on repowise.dev/settings?tab=bot}