eval: correct two question-set expectations found wrong by hand-audit

[swati510]

precision@high was 0.937 — four of 63 confident answers had no expected page in front of them. That figure is only worth quoting once someone has read those four, so I read all of them (five, in fact: the two runs disagree on one question each, so the union is q002, q015, q030, q047, q082).

Two of the five are the question set being wrong, not the tool. Those are corrected here. The other three are genuine and are deliberately left alone.

q002 — one question, two correct files

Where does the repo scan stop itself from descending into nested git repositories?

Expected core/fs_walk.py. The tool answered core/workspace/scanner.py, which genuinely does this — "Stops descending once a .git boundary is found (no nested repos)". One scan walks a repo's files, the other discovers repos in a workspace; both stop at .git, and the question named neither. Reworded to ask for the walk option (prune_nested_git), which only the file walk has.

q015 — marked wrong for being right

Where does get_answer run its retrieval?

Expected tool_answer/retrieval.py, whose own docstring opens: "These operate on the candidate hit list after the hybrid-retrieval stages in _answer_pipeline". The tool cited _answer_pipeline.py. It was correct and scored zero. Reworded to ask for what the expected module actually does — the domain penalty and intersection boost.

Left alone as genuine misses

• q030 — "how is a file assigned to a layer when the knowledge graph has none for it". generation/layers.py::infer_layer says verbatim it is "the fallback used when the knowledge graph has no layer for a file". The tool cited tool_context/kg.py::_find_layer_for_file, which only looks a file up in the graph's layers, and hedged in the prose ("presumably", "the full source body is not provided") while still returning high in one run.
• q047 — asked why a config-looking file is protected from the deletion-ready tier; the tool cited two actual config files and explained that they are entry points, rather than dead_code/risk_factors.py where the rule lives. Adjacent-but-wrong page, high confidence.
• q082 — the answer names the expected file in its prose, with the right symbol, and cites two others. Retrieval found it; citation dropped it. The counter added in eval: keep the answer text, and count pages named in prose but not cited #28 is what makes this visible without a re-run; its "blocked directories" half is separately wrong, pointing at dead-code path globs rather than LanguageRegistry.blocked_dirs.

Tests. Both corrections are pinned by a parametrised test carrying the reason next to the corrected text, so a later edit cannot quietly restore either original. Reverting only the JSONL fails both; with it, tests/test_answer_eval_*.py → 133 passed.

[repowise-bot]

✅ Health: 9.9 (unchanged)

📋 At a glance
1 hotspot touched. Scoped to tests.

🔎 More signals (1)

🔥 Hotspot touched (1)

• tests/test_answer_eval_questions_file.py — 1 commits/90d, 0 dependents · primary owner: Swati Ahuja (100%)

---

📊 Full report · ⭐ Star Repowise · 📥 Install bot · Last updated 2026-07-29 17:35 UTC
_{Silence on a single PR with [skip repowise] in the title · Per-repo toggle on repowise.dev/settings?tab=bot}