The seal gate never checked the four hashes it exists to check - #2746
Merged
Conversation
`check_seal_coverage.py` is named "Every seal still describes its spec" and its
own docstring says a seal is broken when "gen_hashes no longer describe what it
produces". It compared `spec_hash` and stopped. Two controls:
spec_hash := zeros -> exit 1, reported stale
gen_hash_zig := zeros -> exit 0, SILENT
So a seal could assert false output for as long as nobody touched the spec --
which is most of them.
MEASURED, by recomputing all five hashes for every seal:
seals the gate called broken 418
seals actually not describing their output 1,078
of which spec AND output drifted 460 (the gate saw these)
of which ONLY THE OUTPUT drifted 612 (invisible to it)
of which only the spec drifted 6
The 612 are the blind spot. They passed every run of this gate while naming
output the compiler had stopped producing.
Three changes:
1. Every seal is re-sealed from what `t27c seal` produces today -- all four
gen_hashes, not just spec_hash. 1,078 files.
2. The gate recomputes the gen_hashes and reports `gen-drift`. Because every
seal was just re-sealed, this lands GREEN: 1,316 seals, 1,222 hold, 94
known-broken. From here any drift is caught on the commit that causes it.
3. seal_baseline.txt drops the 55 entries that now hold -- the gate had been
asking for this in a NOTE -- 151 lines to 96. One dangling seal is removed:
vscode-trinity-swe/test_highlight.t27 was deleted in b79702e, and the
gate's own prescription for dangling is "remove the seal with it".
Recomputing needs the compiler, so seal-coverage.yml now builds it, and a
missing binary exits 2 with a message instead of returning a pass it did not
earn. The self-check's controls plant synthetic trees with no compiler; they set
T27_SEAL_SKIP_GEN explicitly rather than letting the script guess from its
surroundings, because a guess would also disable the check in a real checkout
where the build merely failed.
Controls, all four re-run after the change:
gen_hash corrupted -> exit 1, gen-drift (was exit 0)
spec_hash corrupted -> exit 1, stale
compiler absent -> exit 2, says so (was a pass)
clean tree -> exit 0
--self-check -> exit 0
Refs #2161
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Refs #2161 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-28 00:39:25 UTC
Summary
Seal Status
|
This was referenced Aug 28, 2026
gHashTag
added a commit
that referenced
this pull request
Aug 28, 2026
* fix(seals): answer "nothing to check" before "the tool is missing" #2746 put the compiler requirement first, so an empty tree got check_seal_coverage: the compiler is not built which is true and is not the diagnosis. With no seals there is nothing to recompute, and the answer the file already had is the right one: FAIL: no seals found at all -- the path is wrong, not the tree check_gate_preconditions.py hands every gate in its table an empty tree and asks what it says. It caught this on the commit after #2746 -- a gate doing exactly its job, one merge after I added a way for this one to lie. Controls, all re-run: empty tree -> exit 1, "no seals found at all" gen_hash corrupted -> exit 1, gen-drift compiler absent -> exit 2, says so clean tree -> exit 0 --self-check -> exit 0 check_gate_preconditions.py and its --self-check -> exit 0 Refs #2161 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(now): ordering fix in the seal gate Refs #2161 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #2161
tools/check_seal_coverage.pyruns under the step name "Every seal still describes its spec", and its own docstring says a seal is broken when "gen_hashes no longer describe what it produces".It compared
spec_hashand stopped.A seal could assert false output for as long as nobody touched the spec — which is most of them.
How much it was missing
Recomputing all five hashes for every seal with
t27c seal:The 612 passed every run of this gate while naming output the compiler had stopped producing.
Three changes
t27c sealproduces today — all fourgen_hashes, not justspec_hash. 1,078 files.gen-drift. Because every seal was just re-sealed, this lands green: 1,316 seals, 1,222 hold, 94 known-broken. From here, drift is caught on the commit that causes it rather than accumulating for a year.seal_baseline.txtdrops the 55 entries that now hold — the gate had been asking for exactly this in a NOTE every run — 151 lines to 96. One dangling seal is removed:vscode-trinity-swe/test_highlight.t27was deleted inb79702ee1, and the gate's own prescription for dangling is "remove the seal with it".Recomputing needs the compiler, so
seal-coverage.ymlbuilds it. A missing binary exits 2 and says so rather than returning a pass it did not earn.The self-check's controls plant synthetic trees with no compiler, so they set
T27_SEAL_SKIP_GENexplicitly rather than letting the script guess from its surroundings — a guess would also switch the check off in a real checkout where the build merely failed.Controls, all re-run after the change
Note on the diff
1,088 files, and 6,200 insertions against 6,172 deletions — almost all of it is five hash lines and a
sealed_atper seal. The reviewable part istools/check_seal_coverage.py,tools/seal_baseline.txtand.github/workflows/seal-coverage.yml.This branch is rebased onto #2745, whose emitter change moves the output of 10 specs; the seals here were computed against that compiler, and the gate is green on the rebased tree.
🤖 Generated with Claude Code