Skip to content

The corpus Zig column cannot see a defect inside a function body, so no ratchet can go red for one #2966

Description

@gHashTag

The column could not see the repair

Yesterday a change unblocked 32 specs under zig test --test-no-exec and moved
t27c corpus's Zig column by exactly 0 (#2952, PR #2963).

That zero is correct. zig build-obj -fno-emit-bin, which the corpus table runs, resolves
identifiers and never Sema-analyses a function nothing references — so a defect inside a
body is invisible to it, and stays invisible when it is repaired.

The consequence is not a reporting quibble:

  • no ratchet can go red for a Zig defect inside a function body, because the number a
    ratchet would watch does not move for one;
  • no repair of that class can be credited, so the report of a real fix reads as a fix
    that did nothing;
  • and the gap is not small — 282 accept under build-obj, 165 under the deeper ruler.
    117 files are counted as Zig-accepted today while carrying something the compiler would
    reject the moment it looked.

The change

A second Zig row, measured with zig test --test-no-exec -femit-bin=<path>, printed beside
the first and carried into --json and --per-spec.

  generates Zig                 31   77.5%
    ... and Zig accepts it      12   30.0%
    ... and Zig ANALYSES it     11   27.5%
  • JSON gains "zig_bodies", because wave-to-wave comparison reads the JSON and a row
    printed only in the human table is a number nothing can diff.
  • --per-spec Zig field goes from two digits to three, header renamed
    zig(gen,build,bodies). That file is what an actual before/after diff is taken over, and
    a reader keyed to two digits would silently drop the new one.

-femit-bin is given a real path. -femit-bin=/dev/null fails on every input with
"failed to invalidate kernel cache: PermissionDenied" — it would report 0 of 589 and read
as a catastrophic regression rather than as a broken ruler. That trap cost a re-run while
measuring #2952 by hand and is written into the code beside the flag.

Tests, and the invariant they pin

Three, deliberately cheap (small --limit), asserting the shape and the invariant
rather than any count — counts move every time the emitter improves, and a test pinned to
one gets re-blessed rather than read.

test mutant that kills it
bodies ⊆ build ⊆ gen o.zig_bodies = true
the column reaches the JSON removing the field from the JSON line
--per-spec carries three digits and says so in its header reverting the format to two

The subset invariant is the load-bearing one: a file the deeper ruler accepts must pass
the shallower one, because analysing a declaration is strictly more work than resolving its
name. A violation means one of the two rulers is broken, not that the compiler improved.

Measured before asserting it: it holds on a 150-spec sample (zig_gen 131, zig_build 35,
zig_bodies 28, 0 timeouts).

When zig is not on PATH the tests print SKIPPED and say so, rather than passing
silently — a green that means "not run" is the thing this whole change is about.

Cost

One extra zig test --test-no-exec per generating spec. The three tests take ~50 s
together at their limits.

A population correction that belongs here

A parallel sweep re-derived the corpus population and it is 665 specs, not the 650 that
several reports — mine included — have been quoting, and cc acceptance is 268 of 589
generating
, not 268 of 650. The numerator was right in every report; the denominator was
wrong in all of them.

Related: /tmp/specs.txt was regenerated by a background agent between the moment a
measurement printed its population and the moment it used it, which is how a 649-entry list
from the previous day became 665 mid-session. The A/B survived because both sides ran after
the change — luck, not design. Written up as skill 401.

Refs #2952

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions