eval: run the judged gold set, and meter what synthesis actually costs

[swati510]

Merge after #30 and #31. Based on master and complete on its own — it carries those two as one squashed base commit — but only the second commit here is new.

Wires the gold set and the judge into a run: python -m answer_eval --gold --index <id> --checkout <tree> --commit <sha>. Same index, same server, one judged blob.

Deliberately a separate runner from the retrieval one. A retrieval run is free and repeatable; a judged run costs two model calls per question. Merging them would make the cheap measurement expensive.

What the blob carries that a correctness number would hide

• correctness_by_confidence — the tool's confidence is a promise to the caller that verification is unnecessary. A single number cannot say whether it is kept; only the breakdown can.
• n_contradicted_at_high — a confident wrong mechanism and a thin-but-honest answer both score badly, and only this separates them. They need opposite fixes.
• found_expected_path per question — separates "retrieval missed it" from "retrieval found it and synthesis got it wrong". Recorded, never scored: a right answer citing an unexpected page is still right.

Metering the tool's own spend

The answer tool has no token accounting — nothing captures usage, no cost row is written anywhere, and every cost figure quoted about it so far is inferred from the agent side. token_meter wraps the provider the registry hands back for the length of a run and adds up what it reports, priced from the product's own table so an eval figure and a product figure cannot drift apart.

Two ways that could quietly report nothing are refused rather than reported: a run that answered questions while metering zero calls (which would mean the tool stopped resolving through the registry, and every cost is silently $0.00), and a model the price table does not know (same result, different cause). A genuinely free provider is still allowed to cost zero.

The field is tool_side_usd_per_question and not $/q, because it is not the cache-neutral cost a caller pays — that is an agent-side quantity a different harness measures, and the two must not be read as one number.

Tests

27 new, both the tool and the judge stubbed: the correctness mean, the answer text kept next to its verdict, an abstention recorded as such, the confidence breakdown, a contradiction at high counted on its own and warned about as it happens, retrieval recorded-but-not-scored, cost per question from a metered run, None rather than 0.0 from an unmetered one, judge tokens reported separately, and the empty-set / non-dict / symbol-less-index / same-family-judge cases.

The meter's own: totals across calls, a response with no usage counting as zero rather than crashing, cost from the price table, an unpriced model raising, a free provider costing zero without raising, the registry restored afterwards, double-wrapping not double-counting, metering nothing raising, and a multi-model meter warning.

python -m pytest tests/test_answer_eval_*.py → 206 passed.

First run

correctness=0.414 · fully-correct 2/22 · contradicted 0 · by confidence {high 0.35, low 0.75, medium 0.55} · tool-side $0.0012/q

I read the zero-scoring answers by hand before trusting the judge, and it errs generous rather than pedantic. Findings are in the plan's log, not here.

[repowise-bot]

✅ Health: 9.9 (unchanged)

📋 At a glance
1 file changed health · 1 hotspot touched · 10 new findings introduced. Scoped to answer_eval.

🚨 Change risk: high _{(riskier than 84% of this repo's commits)}
This change's risk is driven by:

• more lines added than baseline
• more scattered than baseline

File	Score	Δ	Why
answer_eval/__main__.py	9.6 → 8.5	▼ -1.1	🔻 introduced complex method, primitive obsession

🔎 More signals (1)

🔥 Hotspot touched (1)

• answer_eval/__main__.py — 2 commits/90d, 0 dependents · primary owner: Swati Ahuja (100%)

---

📊 Full report · ⭐ Star Repowise · 📥 Install bot · Last updated 2026-07-29 17:53 UTC
_{Silence on a single PR with [skip repowise] in the title · Per-repo toggle on repowise.dev/settings?tab=bot}