Conversation
Runs the six issue-triage cases against any model opencode can reach, with a with_skill/without_skill arm each. The prompts differ from evals.json out of necessity -- opencode resolves slash commands from .opencode/command/ rather than from skills, so /issue-triage becomes a skill-tool instruction -- which makes these results non-comparable to the Agent-tool records without reading the harness_deltas field first. Fixture paths are absolute because relative ones are dangerous here: a baseline run resolved evals/fixtures/issue-307.json against $HOME, found nothing, and webfetched the live issue instead, grading against today's thread rather than the frozen report with no sign in the output. webfetch is now denied and the runner refuses to start a run naming a path that does not exist. check_no_writes.py needs a distilled tool-call log rather than the transcript: opencode's skill tool inlines SKILL.md, which contains 'issue_write' in the sentence forbidding it, so every with-skill run failed on the skill's own prohibition. big-pickle at variant=high scored with_skill 29/42, without_skill 25/42. The totals understate what separates them (evidence quality, and never drafting a close) and hide two inversions where the baseline scored higher. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every prior record in this directory ran with_skill only, so none of them could
answer whether the skill helps -- they compared models to each other. This adds
the missing baseline arm: 6 cases x {opus-5 medium, sonnet-5 high} x {with, without}.
Effort is set per agent rather than inherited, which is a first for these
records; both earlier files carry session-default effort.
Result: opus 38/42 with vs 33/42 without, sonnet 33/42 with vs 34/42 without.
The whole delta is case 1 (12 -> 19 summed across models); cases 2-6 move 46 ->
45. The skill supplies procedure, not reasoning -- bare Opus analysed the due_at
bug better than the assertion list can measure, then skipped the duplicate
search, the regression test and the labels entirely.
Cases 4, 5 and 6 now score identically across all four Claude cells. By the
standard in results/README.md they have stopped discriminating at this tier.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
skill-creator is the off-the-shelf harness these evals were written for -- same evals.json schema, same with_skill/without_skill workspace layout, and it ships aggregation with variance, a grader, and an analyzer that looks for exactly the non-discriminating assertions we found by hand. Regrading the 12 opus runs with its grader.md agrees with the hand grading in 10 of 12 cells; both disagreements are +1 and land one per arm, so the skill delta is +5 either way. The scores stand. Its value is grader.md steps 4 and 6, which the hand grading skipped. Two rubric defects fell out: - Case 3's version-floor assertion is unsatisfiable. fixtures/issue-270.json contains no version information at all, and 8 of 8 runs across three models and both arms have failed it. That is a broken assertion, not a model weakness. - Case 1 is partly teaching to the test. references/repro-harness.md works through issue 307 by name, including the RedBeatCase subclass, the expectedFailure marker and the tests/test_entry.py destination -- three of the ten assertions. Case 1 is also the only case where the skill measurably helps on Claude models, so part of that gain is transcription from a file the baseline never sees. Step 4 claim-checking also caught a wrong sha, a missed commit, a line ref off by six and an overstated reproduction claim inside a run that scored 10/10. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Case 3's first assertion demanded that the run note the reported environment predates the supported celery/python floor. fixtures/issue-270.json states no version at all -- the only temporal signal is the 2024-02-02 filing date, which is already above setup.cfg's floor. It had been failed by 8 of 8 runs across three models and both arms. Rewritten around what the fixture actually supports; it passed for the first time on the with_skill re-run. Case 1's label assertion was negative-only, so a run proposing no labels scored the same as one proposing the right ones. Rewritten as positive plus guard; the isolated baseline now correctly fails it for proposing 'good first issue'. Case 1 also gains an assertion_kinds field marking three assertions transcribable: references/repro-harness.md works through issue 307 by name, so the RedBeatCase subclass, the expectedFailure marker and the tests/test_entry.py destination are copyable by any run that reads the skill. The isolated re-run confirms the tag is accurate -- 3 of 3 pass with the skill, 0 of 3 without, and they do not leak from the repo's own test suite. Isolated results, opus medium: case 1 10/10 vs 4/10, case 3 6/6 vs 5/6. Excluding the transcribable group, case 1 is 7/7 vs 4/7. This is the first record here whose baseline was actually isolated. A first attempt without worktrees showed why it matters: both baselines read SKILL.md and the references off disk, and the two arms collided on the same scratch filename in the shared tree. Prompt omission is not enforcement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- big-pickle called #307 probably-fixed on the strength of an unmerged branch, which would have told the reporter their live bug was handled. Add an explicit rule: a fix has to be an ancestor of main before it counts, an unmerged fix is reported as such, not as fixed. - one run's dry-run draft comment read 'I've added a failing regression test in #PPP' -- a placeholder claiming an action that was never taken. State the tense explicitly: a dry-run draft describes work actually done, never narrates a comment/branch/PR that wasn't created. - the duplicate-search example named issue #210 by number, which is also this skill's own eval fixture -- a with_skill run got the answer read straight out of the guidance meant to teach it how to search. Genericised the example so it teaches the technique without pre-solving the case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
opencode/big-pickle.fixtures/issue-270.jsonnever states — failed by 8 of 8 runs across three models and both arms.main.#PPPplaceholder) — now states explicitly that dry-run drafts describe work actually done, never narrate actions that weren't taken.#210) that's also this suite's own eval fixture, handing a with-skill run the answer to its own grading. Genericised.Why
The skill's actual measured benefit over a bare model was much smaller than the first pass suggested (see
evals/results/2026-07-27-isolated-cases-1-3.json) — a meaningful chunk of the original delta came from a broken assertion and from a reference file that transcribes the answer to its own eval case. Chasing that down surfaced real bugs in the skill itself, which are the more valuable fix here.What a reviewer should know
evals/results/*.jsonare point-in-time measurement records, not code — several were produced, then partially superseded by later ones in the same session as isolation/rubric bugs were found.2026-07-27-isolated-cases-1-3.jsonand the SKILL.md fixes are the parts that matter going forward; the intermediate records are kept for the paper trail (see each file's own caveats)..claude/skills/issue-triage/(the skill definition, its eval harness, and eval result records).evals/opencode/README.mddocuments the new harness and its known gaps (unblinded grading, no repeated sampling, self-reported vs trace-verified execution).