Skip to content

The seal gate never checked the four hashes it exists to check - #2746

Merged
gHashTag merged 2 commits into
masterfrom
w699-seal-gate
Aug 28, 2026
Merged

The seal gate never checked the four hashes it exists to check#2746
gHashTag merged 2 commits into
masterfrom
w699-seal-gate

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Refs #2161

tools/check_seal_coverage.py runs under the step name "Every seal still describes its spec", and its own docstring says a seal is broken when "gen_hashes no longer describe what it produces".

It compared spec_hash and stopped.

spec_hash    := zeros   ->  exit 1, reported stale
gen_hash_zig := zeros   ->  exit 0, SILENT

A seal could assert false output for as long as nobody touched the spec — which is most of them.

How much it was missing

Recomputing all five hashes for every seal with t27c seal:

seals
the gate called broken 418
actually not describing their output 1,078
· spec and output drifted 460 — the gate saw these
· only the output drifted 612 — invisible to it
· only the spec drifted 6

The 612 passed every run of this gate while naming output the compiler had stopped producing.

Three changes

  1. Every seal re-sealed from what t27c seal produces today — all four gen_hashes, not just spec_hash. 1,078 files.
  2. The gate recomputes them and reports gen-drift. Because every seal was just re-sealed, this lands green: 1,316 seals, 1,222 hold, 94 known-broken. From here, drift is caught on the commit that causes it rather than accumulating for a year.
  3. seal_baseline.txt drops the 55 entries that now hold — the gate had been asking for exactly this in a NOTE every run — 151 lines to 96. One dangling seal is removed: vscode-trinity-swe/test_highlight.t27 was deleted in b79702ee1, and the gate's own prescription for dangling is "remove the seal with it".

Recomputing needs the compiler, so seal-coverage.yml builds it. A missing binary exits 2 and says so rather than returning a pass it did not earn.

The self-check's controls plant synthetic trees with no compiler, so they set T27_SEAL_SKIP_GEN explicitly rather than letting the script guess from its surroundings — a guess would also switch the check off in a real checkout where the build merely failed.

Controls, all re-run after the change

gen_hash corrupted   -> exit 1, gen-drift     (was exit 0, silent)
spec_hash corrupted  -> exit 1, stale
compiler absent      -> exit 2, says so       (was a pass)
clean tree           -> exit 0
--self-check         -> exit 0

Note on the diff

1,088 files, and 6,200 insertions against 6,172 deletions — almost all of it is five hash lines and a sealed_at per seal. The reviewable part is tools/check_seal_coverage.py, tools/seal_baseline.txt and .github/workflows/seal-coverage.yml.

This branch is rebased onto #2745, whose emitter change moves the output of 10 specs; the seals here were computed against that compiler, and the gate is green on the rebased tree.

🤖 Generated with Claude Code

gHashTag and others added 2 commits August 28, 2026 07:37
`check_seal_coverage.py` is named "Every seal still describes its spec" and its
own docstring says a seal is broken when "gen_hashes no longer describe what it
produces". It compared `spec_hash` and stopped. Two controls:

    spec_hash    := zeros   ->  exit 1, reported stale
    gen_hash_zig := zeros   ->  exit 0, SILENT

So a seal could assert false output for as long as nobody touched the spec --
which is most of them.

MEASURED, by recomputing all five hashes for every seal:

    seals the gate called broken                      418
    seals actually not describing their output      1,078
      of which spec AND output drifted                460   (the gate saw these)
      of which ONLY THE OUTPUT drifted                612   (invisible to it)
      of which only the spec drifted                    6

The 612 are the blind spot. They passed every run of this gate while naming
output the compiler had stopped producing.

Three changes:

1. Every seal is re-sealed from what `t27c seal` produces today -- all four
   gen_hashes, not just spec_hash. 1,078 files.
2. The gate recomputes the gen_hashes and reports `gen-drift`. Because every
   seal was just re-sealed, this lands GREEN: 1,316 seals, 1,222 hold, 94
   known-broken. From here any drift is caught on the commit that causes it.
3. seal_baseline.txt drops the 55 entries that now hold -- the gate had been
   asking for this in a NOTE -- 151 lines to 96. One dangling seal is removed:
   vscode-trinity-swe/test_highlight.t27 was deleted in b79702e, and the
   gate's own prescription for dangling is "remove the seal with it".

Recomputing needs the compiler, so seal-coverage.yml now builds it, and a
missing binary exits 2 with a message instead of returning a pass it did not
earn. The self-check's controls plant synthetic trees with no compiler; they set
T27_SEAL_SKIP_GEN explicitly rather than letting the script guess from its
surroundings, because a guess would also disable the check in a real checkout
where the build merely failed.

Controls, all four re-run after the change:

    gen_hash corrupted     -> exit 1, gen-drift     (was exit 0)
    spec_hash corrupted    -> exit 1, stale
    compiler absent        -> exit 2, says so       (was a pass)
    clean tree             -> exit 0
    --self-check           -> exit 0

Refs #2161

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Refs #2161

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-28 00:39:25 UTC

Summary

Status Count
Total Open PRs 8
PRs with Failing Checks 7
PRs with All Checks Green 1
READY 0
FAILING 7
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=25a9c40dc273 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@gHashTag
gHashTag merged commit fc14fd6 into master Aug 28, 2026
22 of 23 checks passed
@gHashTag
gHashTag deleted the w699-seal-gate branch August 28, 2026 00:50
gHashTag added a commit that referenced this pull request Aug 28, 2026
* fix(seals): answer "nothing to check" before "the tool is missing"

#2746 put the compiler requirement first, so an empty tree got

    check_seal_coverage: the compiler is not built

which is true and is not the diagnosis. With no seals there is nothing to
recompute, and the answer the file already had is the right one:

    FAIL: no seals found at all -- the path is wrong, not the tree

check_gate_preconditions.py hands every gate in its table an empty tree and
asks what it says. It caught this on the commit after #2746 -- a gate doing
exactly its job, one merge after I added a way for this one to lie.

Controls, all re-run:

    empty tree           -> exit 1, "no seals found at all"
    gen_hash corrupted   -> exit 1, gen-drift
    compiler absent      -> exit 2, says so
    clean tree           -> exit 0
    --self-check         -> exit 0
    check_gate_preconditions.py and its --self-check -> exit 0

Refs #2161

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(now): ordering fix in the seal gate

Refs #2161

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant