The column could not see the repair
Yesterday a change unblocked 32 specs under zig test --test-no-exec and moved
t27c corpus's Zig column by exactly 0 (#2952, PR #2963).
That zero is correct. zig build-obj -fno-emit-bin, which the corpus table runs, resolves
identifiers and never Sema-analyses a function nothing references — so a defect inside a
body is invisible to it, and stays invisible when it is repaired.
The consequence is not a reporting quibble:
- no ratchet can go red for a Zig defect inside a function body, because the number a
ratchet would watch does not move for one;
- no repair of that class can be credited, so the report of a real fix reads as a fix
that did nothing;
- and the gap is not small — 282 accept under
build-obj, 165 under the deeper ruler.
117 files are counted as Zig-accepted today while carrying something the compiler would
reject the moment it looked.
The change
A second Zig row, measured with zig test --test-no-exec -femit-bin=<path>, printed beside
the first and carried into --json and --per-spec.
generates Zig 31 77.5%
... and Zig accepts it 12 30.0%
... and Zig ANALYSES it 11 27.5%
- JSON gains
"zig_bodies", because wave-to-wave comparison reads the JSON and a row
printed only in the human table is a number nothing can diff.
--per-spec Zig field goes from two digits to three, header renamed
zig(gen,build,bodies). That file is what an actual before/after diff is taken over, and
a reader keyed to two digits would silently drop the new one.
-femit-bin is given a real path. -femit-bin=/dev/null fails on every input with
"failed to invalidate kernel cache: PermissionDenied" — it would report 0 of 589 and read
as a catastrophic regression rather than as a broken ruler. That trap cost a re-run while
measuring #2952 by hand and is written into the code beside the flag.
Tests, and the invariant they pin
Three, deliberately cheap (small --limit), asserting the shape and the invariant
rather than any count — counts move every time the emitter improves, and a test pinned to
one gets re-blessed rather than read.
| test |
mutant that kills it |
bodies ⊆ build ⊆ gen |
o.zig_bodies = true |
| the column reaches the JSON |
removing the field from the JSON line |
--per-spec carries three digits and says so in its header |
reverting the format to two |
The subset invariant is the load-bearing one: a file the deeper ruler accepts must pass
the shallower one, because analysing a declaration is strictly more work than resolving its
name. A violation means one of the two rulers is broken, not that the compiler improved.
Measured before asserting it: it holds on a 150-spec sample (zig_gen 131, zig_build 35,
zig_bodies 28, 0 timeouts).
When zig is not on PATH the tests print SKIPPED and say so, rather than passing
silently — a green that means "not run" is the thing this whole change is about.
Cost
One extra zig test --test-no-exec per generating spec. The three tests take ~50 s
together at their limits.
A population correction that belongs here
A parallel sweep re-derived the corpus population and it is 665 specs, not the 650 that
several reports — mine included — have been quoting, and cc acceptance is 268 of 589
generating, not 268 of 650. The numerator was right in every report; the denominator was
wrong in all of them.
Related: /tmp/specs.txt was regenerated by a background agent between the moment a
measurement printed its population and the moment it used it, which is how a 649-entry list
from the previous day became 665 mid-session. The A/B survived because both sides ran after
the change — luck, not design. Written up as skill 401.
Refs #2952
The column could not see the repair
Yesterday a change unblocked 32 specs under
zig test --test-no-execand movedt27c corpus's Zig column by exactly 0 (#2952, PR #2963).That zero is correct.
zig build-obj -fno-emit-bin, which the corpus table runs, resolvesidentifiers and never Sema-analyses a function nothing references — so a defect inside a
body is invisible to it, and stays invisible when it is repaired.
The consequence is not a reporting quibble:
ratchet would watch does not move for one;
that did nothing;
build-obj, 165 under the deeper ruler.117 files are counted as Zig-accepted today while carrying something the compiler would
reject the moment it looked.
The change
A second Zig row, measured with
zig test --test-no-exec -femit-bin=<path>, printed besidethe first and carried into
--jsonand--per-spec."zig_bodies", because wave-to-wave comparison reads the JSON and a rowprinted only in the human table is a number nothing can diff.
--per-specZig field goes from two digits to three, header renamedzig(gen,build,bodies). That file is what an actual before/after diff is taken over, anda reader keyed to two digits would silently drop the new one.
-femit-binis given a real path.-femit-bin=/dev/nullfails on every input with"failed to invalidate kernel cache: PermissionDenied" — it would report 0 of 589 and read
as a catastrophic regression rather than as a broken ruler. That trap cost a re-run while
measuring #2952 by hand and is written into the code beside the flag.
Tests, and the invariant they pin
Three, deliberately cheap (small
--limit), asserting the shape and the invariantrather than any count — counts move every time the emitter improves, and a test pinned to
one gets re-blessed rather than read.
bodies ⊆ build ⊆ geno.zig_bodies = true--per-speccarries three digits and says so in its headerThe subset invariant is the load-bearing one: a file the deeper ruler accepts must pass
the shallower one, because analysing a declaration is strictly more work than resolving its
name. A violation means one of the two rulers is broken, not that the compiler improved.
Measured before asserting it: it holds on a 150-spec sample (
zig_gen 131,zig_build 35,zig_bodies 28, 0 timeouts).When
zigis not on PATH the tests print SKIPPED and say so, rather than passingsilently — a green that means "not run" is the thing this whole change is about.
Cost
One extra
zig test --test-no-execper generating spec. The three tests take ~50 stogether at their limits.
A population correction that belongs here
A parallel sweep re-derived the corpus population and it is 665 specs, not the 650 that
several reports — mine included — have been quoting, and
ccacceptance is 268 of 589generating, not 268 of 650. The numerator was right in every report; the denominator was
wrong in all of them.
Related:
/tmp/specs.txtwas regenerated by a background agent between the moment ameasurement printed its population and the moment it used it, which is how a 649-entry list
from the previous day became 665 mid-session. The A/B survived because both sides ran after
the change — luck, not design. Written up as skill 401.
Refs #2952