eval: the 22-question judged gold set and its loader

[swati510]

The retrieval set asks whether the right page reached the caller. It cannot ask whether the answer was right — and that is the failure the confident-answer tail is made of: an answer can cite the correct file and still describe the wrong mechanism. Hand-auditing the confident misses turned up exactly that shape twice.

This is the questions and the loader. Nothing calls a model yet — the judge and the runner follow separately, so this one is pure and green on its own.

Correctness is judged point by point

Each question carries the claims a correct answer has to make, judged one at a time, plus the plausible wrong claims that score it zero however much else it covered.

{"id": "g005",
 "question": "How is a commit classified when an agent service identity is the committer but a human is the author?",
 "must_include": ["it is classified tier 2 (human-driven agent), not tier 1", ...],
 "must_not_claim": ["an agent identity as committer makes the commit tier 1 (near-autonomous)"],
 "expected_paths": ["packages/core/.../agent_provenance.py"]}

A judge asked for a single number rewards fluency, and an answer that gets three mechanisms of four is a different thing from one that gets none — a distinction a holistic grade throws away exactly when the tool is improving.

must_not_claim is the other half, and it is the part aimed at the observed failure. Several of the worst answers were confident, well written, and about the adjacent page; nothing short of naming the tempting wrong claim marks that as wrong rather than merely incomplete. A question whose answer makes one is zero — a wrong mechanism stated confidently is worse than an incomplete answer, and averaging them hides it.

The set

22 questions · 87 required points · 16 carrying a forbidden claim · 21 distinct pages, no page used more than twice. Shapes: 8 what, 7 how, 5 why, 1 where, 1 when.

Each is written from the content of the page it expects, so a reviewer can check any expectation by reading that one page. Every expected path was verified to resolve to a real page in snapshot 45ce57f52457.

Tests

23 new. The loader raises on everything that would make a correctness figure unreliable: a missing file, an unparseable line (reported with its line number), a missing or blank field, an unknown key (a mistyped must_includes is how a required point silently stops being checked), an empty must_include (which would score 100% against any answer, including none), a duplicate id, a non-string point, and a claim listed as both required and forbidden — which could never be scored, since covering it also fails it.

The file's own test guards the properties that decide whether the number means anything: at least three points per question, no point vague enough that a judge marks it covered against any fluent answer, a forbidden claim on at least half, more than one question shape, and no page used more than twice.

That vagueness check earned its place immediately — it failed on "it stays a single group", which I rewrote rather than lowering the threshold.

python -m pytest tests/test_answer_eval_*.py → 154 passed (was 131).

[repowise-bot]

✅ Health: 9.9 (unchanged)

📋 At a glance
2 new findings introduced.

⚠️ Change risk: moderate _{(riskier than 49% of this repo's commits)}
This change's risk is driven by:

• more lines added than baseline
• more scattered than baseline

---

📊 Full report · ⭐ Star Repowise · 📥 Install bot · Last updated 2026-07-29 17:42 UTC
_{Silence on a single PR with [skip repowise] in the title · Per-repo toggle on repowise.dev/settings?tab=bot}