corpus: report all four backends, not two (Refs #2161) - #2738
Merged
Conversation
`corpus` describes itself as "the only corpus metric that does not lie"
and measured Zig and Verilog. The Rust backend had no compile gate
anywhere in this repository and neither did C -- which is how an empty
`match` for every `switch`, a dropped body for every `for`, and a `u64`
typed as `int` all shipped with a green exit.
Two stages added, mirroring the existing Zig block:
gen-rust -> rustc --edition 2021 --crate-type lib --emit=metadata
gen-c -> cc -fsyntax-only -std=gnu11
and one row the two-backend table could not show: how many specs satisfy
ALL FOUR toolchains. That is what "one spec, four targets" claims, and
until now nothing counted it.
On a 39-spec sample:
generates Zig 22 ... and Zig accepts it 12 30.8%
generates Rust 22 ... and rustc accepts it 0 0.0%
generates C 22 ... and cc accepts it 1 2.6%
generates Verilog 22 ... and iverilog accepts 7 17.9%
Zig AND Verilog accept 6 15.4%
ALL FOUR accept 0 0.0%
The `BOTH backends accept` label is now `Zig AND Verilog accept`, because
with four columns "both" no longer names anything.
Zero is the number this change exists to print. It is a sample, not the
corpus, and the row says so by carrying its own denominator.
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
Contributor
PR DashboardGenerated at: 2026-08-27 19:49:49 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
Contributor
PR DashboardGenerated at: 2026-08-27 20:07:27 UTC
Summary
Seal Status
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
corpusdescribes itself as "the only corpus metric that does not lie" and measured two of the four backends. The Rust backend had no compile gate anywhere in this repository, and neither did C — which is how an emptymatchfor everyswitch, a dropped body for everyfor, and au64typed asintall shipped with a green exit.Two stages added, mirroring the existing Zig block:
and one row the two-backend table could not show: how many specs satisfy all four toolchains. That is what "one spec, four targets" claims, and until now nothing counted it.
On a 39-spec sample
Zero is the number this change exists to print.
It is a sample, not the corpus, and the row carries its own denominator so it says so.
The
BOTH backends acceptlabel is nowZig AND Verilog accept— with four columns the word both no longer names anything, and leaving it would have made the new rows read as a subset of it.Refs #2161