The Rust column measured the ruler: 0 of 559 was never true - #2760
Merged
Conversation
… 559 was never true
`t27c corpus` compiled each generated Rust file with
rustc --edition 2021 --crate-type lib --emit=metadata -A warnings -o /dev/null <file>
and rustc writes its metadata through a temp file NEXT TO the output path. With
`-o /dev/null` it tries to create `/dev/rmeta<random>` and dies:
error: couldn't create a temp dir: Operation not permitted (os error 1)
at path "/dev/rmeta4HIG1c"
On EVERY input. Reproduced here on a 23-character valid Rust file: it fails with
`-o /dev/null` and exits 0 with `-o <path>`. rustc 1.98.0 is on PATH, so this
was never a missing toolchain.
So the Rust column has read zero for as long as it has existed, and that zero
was reported as a finding -- by me, in this session, including to the owner and
in a release note. Corrected:
was is
rustc accepts it 0 144 (22.2%)
ALL FOUR accept 0 60 (9.2%)
Nothing about the emitted Rust changed in this commit. The other three columns
are unmoved -- Zig 217, cc 157, Zig-AND-Verilog 194 -- which is the control:
a change to the rustc invocation should move the rustc number and nothing else.
The lesson is the one this repository already has a name for. Every other
backend's check writes to a real file; this one wrote to /dev/null and its
failure looked exactly like "the generated code is bad". A measurement that
cannot succeed is not a measurement of the thing it names.
Refs #2754
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…the Rust column
Every generated struct carried
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
unconditionally, and the corpus compiles that output as a standalone
`--crate-type lib` with no `--extern`. So every struct-bearing spec failed with
error[E0433]: cannot find module or crate `serde`
Measured on this tree with the ruler fixed in the previous commit: the FIRST
error in 23 of 38 sampled specs. Behind a `cfg_attr` the default output compiles
against std alone, and anyone who wants serialisation turns the feature on and
gets byte-for-byte what was emitted before.
before after
rustc accepts it 144 173 (+29)
ALL FOUR accept 60 63 (+3)
Zig accepts it 217 217
cc accepts it 157 157
Zig and C are the control: a change to the Rust emitter must move the Rust
numbers and nothing else, and it did.
668 seals re-sealed. Worth saying which category caught them: `gen-drift`, added
this morning. Until today this gate compared `spec_hash` only, so a change that
altered generated output while leaving every spec untouched -- exactly this one
-- would have passed it without a word. The gate caught its author's own change
on the commit that made it.
Suite 2424 passed / 0 failed.
Refs #2754
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Refs #2754 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-28 18:33:14 UTC
Summary
Seal Status
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #2754
t27c corpuscompiled each generated Rust file withand rustc writes its metadata through a temp file next to the output path. With
-o /dev/nullit tries to create/dev/rmeta<random>and dies:On every input. Reproduced here on a 23-character valid Rust file: it fails with
-o /dev/nulland exits 0 with-o <path>.rustc 1.98.0is on PATH, so this was never a missing toolchain.The Rust column has therefore read zero for as long as it has existed — and that zero was reported as a finding, by me, in this session, including to the owner and in a release note.
Two commits, two different things
1. The ruler. Nothing about the emitted Rust changes:
2. The largest real defect. Every generated struct carried
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]unconditionally, and the corpus compiles standalone with no--extern— so every struct-bearing spec failed withcannot find module or crate \serde`. Measured with the ruler fixed: the **first** error in 23 of 38 sampled specs. Behind acfg_attr` the default output compiles against std alone, and turning the feature on gives byte-for-byte what was emitted before.Zig and C are the control: a change to the Rust emitter must move the Rust numbers and nothing else, and it did.
The gate caught its author
668 seals re-sealed — under the category
gen-drift, added this morning in #2746. Until today this gate comparedspec_hashonly, so a change that alters generated output while leaving every spec untouched — exactly this one — would have passed it without a word.Suite 2424 passed / 0 failed; seal gate, specs-generate gate and
check_gate_preconditions.pyall exit 0.🤖 Generated with Claude Code